Note: This is test shard 1 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from AdminCliTest
[ RUN ] AdminCliTest.RebalancerReportOnly
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 01:58:45.275736 398 test_util.cc:268] Using random seed: 256857155
I20240503 01:58:45.282718 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 01:58:45.282828 398 ts_itest-base.cc:113] --------------
I20240503 01:58:45.282860 398 ts_itest-base.cc:114] 5 tablet servers
I20240503 01:58:45.282881 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 01:58:45.282894 398 ts_itest-base.cc:116] --------------
2024-05-03T01:58:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T01:58:45Z Disabled control of system clock
I20240503 01:58:45.293131 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:44819
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:35489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:44819 with env {}
W20240503 01:58:45.412426 410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:45.412665 410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:45.412744 410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:45.415830 410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 01:58:45.415891 410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:45.415922 410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 01:58:45.415954 410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 01:58:45.419600 410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:44819
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:44819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:45.420508 410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:45.421664 410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:45.430148 415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:45.430913 416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:45.431794 410 server_base.cc:1034] running on GCE node
W20240503 01:58:45.432715 418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:45.433303 410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:45.434335 410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:45.435494 410 hybrid_clock.cc:648] HybridClock initialized: now 1714701525435455 us; error 56 us; skew 500 ppm
I20240503 01:58:45.441735 410 webserver.cc:460] Webserver started at http://127.0.99.190:46067/ using document root <none> and password file <none>
I20240503 01:58:45.442386 410 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:45.442457 410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:45.442684 410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:45.444509 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d1adc955a9ec4bd3ad83929bd30ff403"
format_stamp: "Formatted at 2024-05-03 01:58:45 on dist-test-slave-w9tw"
I20240503 01:58:45.445013 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d1adc955a9ec4bd3ad83929bd30ff403"
format_stamp: "Formatted at 2024-05-03 01:58:45 on dist-test-slave-w9tw"
I20240503 01:58:45.449193 410 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 01:58:45.452250 424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:45.453642 410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:45.453799 410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d1adc955a9ec4bd3ad83929bd30ff403"
format_stamp: "Formatted at 2024-05-03 01:58:45 on dist-test-slave-w9tw"
I20240503 01:58:45.453913 410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:45.479286 410 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:45.480036 410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:45.480237 410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:45.488201 476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:44819 every 8 connection(s)
I20240503 01:58:45.488202 410 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:44819
I20240503 01:58:45.489318 410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 01:58:45.491506 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 410
I20240503 01:58:45.491649 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 01:58:45.492141 477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:45.497905 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1adc955a9ec4bd3ad83929bd30ff403: Bootstrap starting.
I20240503 01:58:45.500099 477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1adc955a9ec4bd3ad83929bd30ff403: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:45.500905 477 log.cc:824] T 00000000000000000000000000000000 P d1adc955a9ec4bd3ad83929bd30ff403: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:45.502797 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1adc955a9ec4bd3ad83929bd30ff403: No bootstrap required, opened a new log
I20240503 01:58:45.505895 477 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1adc955a9ec4bd3ad83929bd30ff403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1adc955a9ec4bd3ad83929bd30ff403" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44819 } }
I20240503 01:58:45.506108 477 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1adc955a9ec4bd3ad83929bd30ff403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:45.506167 477 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1adc955a9ec4bd3ad83929bd30ff403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1adc955a9ec4bd3ad83929bd30ff403, State: Initialized, Role: FOLLOWER
I20240503 01:58:45.506628 477 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1adc955a9ec4bd3ad83929bd30ff403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1adc955a9ec4bd3ad83929bd30ff403" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44819 } }
I20240503 01:58:45.506775 477 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1adc955a9ec4bd3ad83929bd30ff403 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:58:45.506836 477 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1adc955a9ec4bd3ad83929bd30ff403 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:58:45.506942 477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1adc955a9ec4bd3ad83929bd30ff403 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:45.545977 477 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1adc955a9ec4bd3ad83929bd30ff403 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1adc955a9ec4bd3ad83929bd30ff403" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44819 } }
I20240503 01:58:45.546445 477 leader_election.cc:302] T 00000000000000000000000000000000 P d1adc955a9ec4bd3ad83929bd30ff403 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1adc955a9ec4bd3ad83929bd30ff403; no voters:
I20240503 01:58:45.546820 477 leader_election.cc:288] T 00000000000000000000000000000000 P d1adc955a9ec4bd3ad83929bd30ff403 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:58:45.547827 482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1adc955a9ec4bd3ad83929bd30ff403 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:45.548081 482 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1adc955a9ec4bd3ad83929bd30ff403 [term 1 LEADER]: Becoming Leader. State: Replica: d1adc955a9ec4bd3ad83929bd30ff403, State: Running, Role: LEADER
I20240503 01:58:45.548185 477 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1adc955a9ec4bd3ad83929bd30ff403 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 01:58:45.548444 482 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1adc955a9ec4bd3ad83929bd30ff403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1adc955a9ec4bd3ad83929bd30ff403" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44819 } }
I20240503 01:58:45.550205 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1adc955a9ec4bd3ad83929bd30ff403 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1adc955a9ec4bd3ad83929bd30ff403. Latest consensus state: current_term: 1 leader_uuid: "d1adc955a9ec4bd3ad83929bd30ff403" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1adc955a9ec4bd3ad83929bd30ff403" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44819 } } }
I20240503 01:58:45.550354 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1adc955a9ec4bd3ad83929bd30ff403 [sys.catalog]: This master's current role is: LEADER
I20240503 01:58:45.550293 483 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1adc955a9ec4bd3ad83929bd30ff403 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1adc955a9ec4bd3ad83929bd30ff403" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1adc955a9ec4bd3ad83929bd30ff403" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 44819 } } }
I20240503 01:58:45.550412 483 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1adc955a9ec4bd3ad83929bd30ff403 [sys.catalog]: This master's current role is: LEADER
I20240503 01:58:45.550930 497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 01:58:45.553519 497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 01:58:45.559222 497 catalog_manager.cc:1347] Generated new cluster ID: 2e763a430a2e47f9a91303a8453afb26
I20240503 01:58:45.559309 497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 01:58:45.568583 497 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 01:58:45.569813 497 catalog_manager.cc:1504] Loading token signing keys...
I20240503 01:58:45.577720 497 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d1adc955a9ec4bd3ad83929bd30ff403: Generated new TSK 0
I20240503 01:58:45.578694 497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 01:58:45.587149 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44819
--builtin_ntp_servers=127.0.99.148:35489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240503 01:58:45.695442 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:45.695690 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:45.695735 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:45.699050 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:45.699173 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 01:58:45.702986 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44819
--never_fsync=true
--heap_profile_path=/tmp/kudu.501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:45.703902 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:45.705003 501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:45.713862 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:45.714676 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:45.715421 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:45.716197 501 server_base.cc:1034] running on GCE node
I20240503 01:58:45.716624 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:45.717308 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:45.718470 501 hybrid_clock.cc:648] HybridClock initialized: now 1714701525718444 us; error 39 us; skew 500 ppm
I20240503 01:58:45.720640 501 webserver.cc:460] Webserver started at http://127.0.99.129:34719/ using document root <none> and password file <none>
I20240503 01:58:45.721323 501 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:45.721393 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:45.721586 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:45.723781 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e49e914485594fb991f0bc3e21c3e91c"
format_stamp: "Formatted at 2024-05-03 01:58:45 on dist-test-slave-w9tw"
I20240503 01:58:45.724354 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e49e914485594fb991f0bc3e21c3e91c"
format_stamp: "Formatted at 2024-05-03 01:58:45 on dist-test-slave-w9tw"
I20240503 01:58:45.728327 501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20240503 01:58:45.730549 515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:45.731578 501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:45.731683 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e49e914485594fb991f0bc3e21c3e91c"
format_stamp: "Formatted at 2024-05-03 01:58:45 on dist-test-slave-w9tw"
I20240503 01:58:45.731787 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:45.748066 501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:45.748770 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:45.748943 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:45.749518 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:45.750470 501 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:45.750562 501 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:45.750639 501 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:45.750672 501 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:45.760964 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:37877
I20240503 01:58:45.761008 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:37877 every 8 connection(s)
I20240503 01:58:45.761952 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 01:58:45.762964 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 501
I20240503 01:58:45.763068 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 01:58:45.765424 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44819
--builtin_ntp_servers=127.0.99.148:35489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 01:58:45.774044 629 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44819
I20240503 01:58:45.774310 629 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:45.774948 629 heartbeater.cc:511] Master 127.0.99.190:44819 requested a full tablet report, sending...
I20240503 01:58:45.776712 441 ts_manager.cc:194] Registered new tserver with Master: e49e914485594fb991f0bc3e21c3e91c (127.0.99.129:37877)
I20240503 01:58:45.778789 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:51839
W20240503 01:58:45.880110 632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:45.880383 632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:45.880461 632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:45.883806 632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:45.883940 632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 01:58:45.887833 632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44819
--never_fsync=true
--heap_profile_path=/tmp/kudu.632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:45.888818 632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:45.889958 632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:45.899601 638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:45.900275 637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:45.901072 640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:45.902101 632 server_base.cc:1034] running on GCE node
I20240503 01:58:45.902817 632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:45.903535 632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:45.904757 632 hybrid_clock.cc:648] HybridClock initialized: now 1714701525904728 us; error 55 us; skew 500 ppm
I20240503 01:58:45.907022 632 webserver.cc:460] Webserver started at http://127.0.99.130:37023/ using document root <none> and password file <none>
I20240503 01:58:45.907655 632 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:45.907727 632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:45.907941 632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:45.909842 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "0618dbed286a4cc98e484b9cca9e7904"
format_stamp: "Formatted at 2024-05-03 01:58:45 on dist-test-slave-w9tw"
I20240503 01:58:45.910392 632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "0618dbed286a4cc98e484b9cca9e7904"
format_stamp: "Formatted at 2024-05-03 01:58:45 on dist-test-slave-w9tw"
I20240503 01:58:45.914499 632 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20240503 01:58:45.917009 646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:45.919400 632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240503 01:58:45.919544 632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "0618dbed286a4cc98e484b9cca9e7904"
format_stamp: "Formatted at 2024-05-03 01:58:45 on dist-test-slave-w9tw"
I20240503 01:58:45.919654 632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:45.941871 632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:45.942595 632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:45.942766 632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:45.943421 632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:45.944350 632 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:45.944399 632 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:45.944461 632 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:45.944492 632 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:45.956105 632 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:40971
I20240503 01:58:45.956154 759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:40971 every 8 connection(s)
I20240503 01:58:45.957150 632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 01:58:45.961997 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 632
I20240503 01:58:45.962124 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 01:58:45.964733 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44819
--builtin_ntp_servers=127.0.99.148:35489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 01:58:45.967350 760 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44819
I20240503 01:58:45.967615 760 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:45.968212 760 heartbeater.cc:511] Master 127.0.99.190:44819 requested a full tablet report, sending...
I20240503 01:58:45.969296 441 ts_manager.cc:194] Registered new tserver with Master: 0618dbed286a4cc98e484b9cca9e7904 (127.0.99.130:40971)
I20240503 01:58:45.969859 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:48445
W20240503 01:58:46.098034 763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:46.098284 763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:46.098354 763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:46.101735 763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:46.101907 763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 01:58:46.105980 763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44819
--never_fsync=true
--heap_profile_path=/tmp/kudu.763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:46.107497 763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:46.108754 763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:46.118435 768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:46.119335 771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:46.120363 769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:46.121332 763 server_base.cc:1034] running on GCE node
I20240503 01:58:46.121835 763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:46.122479 763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:46.123697 763 hybrid_clock.cc:648] HybridClock initialized: now 1714701526123669 us; error 48 us; skew 500 ppm
I20240503 01:58:46.125864 763 webserver.cc:460] Webserver started at http://127.0.99.131:39033/ using document root <none> and password file <none>
I20240503 01:58:46.126446 763 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:46.126511 763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:46.126717 763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:46.128407 763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "cccd6ad0cfec4ee8a7c019a220ca91a6"
format_stamp: "Formatted at 2024-05-03 01:58:46 on dist-test-slave-w9tw"
I20240503 01:58:46.128880 763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "cccd6ad0cfec4ee8a7c019a220ca91a6"
format_stamp: "Formatted at 2024-05-03 01:58:46 on dist-test-slave-w9tw"
I20240503 01:58:46.132193 763 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:58:46.134263 777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:46.135407 763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 01:58:46.135525 763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "cccd6ad0cfec4ee8a7c019a220ca91a6"
format_stamp: "Formatted at 2024-05-03 01:58:46 on dist-test-slave-w9tw"
I20240503 01:58:46.135607 763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:46.152838 763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:46.153522 763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:46.153687 763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:46.154239 763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:46.155352 763 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:46.155473 763 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:46.155577 763 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:46.155611 763 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:46.166592 763 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:43117
I20240503 01:58:46.166646 890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:43117 every 8 connection(s)
I20240503 01:58:46.167557 763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 01:58:46.174886 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 763
I20240503 01:58:46.175001 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 01:58:46.177251 891 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44819
I20240503 01:58:46.177508 891 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:46.177642 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44819
--builtin_ntp_servers=127.0.99.148:35489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 01:58:46.178113 891 heartbeater.cc:511] Master 127.0.99.190:44819 requested a full tablet report, sending...
I20240503 01:58:46.179270 441 ts_manager.cc:194] Registered new tserver with Master: cccd6ad0cfec4ee8a7c019a220ca91a6 (127.0.99.131:43117)
I20240503 01:58:46.179905 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:46247
W20240503 01:58:46.290750 894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:46.291013 894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:46.291093 894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:46.294339 894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:46.294533 894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 01:58:46.298579 894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44819
--never_fsync=true
--heap_profile_path=/tmp/kudu.894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:46.299607 894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:46.300791 894 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:46.310340 902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:46.311645 900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:46.312197 899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:46.313232 894 server_base.cc:1034] running on GCE node
I20240503 01:58:46.313732 894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:46.314441 894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:46.315688 894 hybrid_clock.cc:648] HybridClock initialized: now 1714701526315673 us; error 34 us; skew 500 ppm
I20240503 01:58:46.318892 894 webserver.cc:460] Webserver started at http://127.0.99.132:37629/ using document root <none> and password file <none>
I20240503 01:58:46.319697 894 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:46.319790 894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:46.320050 894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:46.322000 894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "805619f602d2475bb1e6dffa668836ef"
format_stamp: "Formatted at 2024-05-03 01:58:46 on dist-test-slave-w9tw"
I20240503 01:58:46.322567 894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "805619f602d2475bb1e6dffa668836ef"
format_stamp: "Formatted at 2024-05-03 01:58:46 on dist-test-slave-w9tw"
I20240503 01:58:46.326509 894 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20240503 01:58:46.329210 908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:46.330662 894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:58:46.330780 894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "805619f602d2475bb1e6dffa668836ef"
format_stamp: "Formatted at 2024-05-03 01:58:46 on dist-test-slave-w9tw"
I20240503 01:58:46.330878 894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:46.343343 894 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:46.343992 894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:46.344138 894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:46.344812 894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:46.345746 894 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:46.345810 894 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:46.345873 894 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:46.345902 894 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:46.357208 894 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:39835
I20240503 01:58:46.357232 1021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:39835 every 8 connection(s)
I20240503 01:58:46.358306 894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 01:58:46.364838 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 894
I20240503 01:58:46.365015 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 01:58:46.368188 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44819
--builtin_ntp_servers=127.0.99.148:35489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 01:58:46.369872 1022 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44819
I20240503 01:58:46.370235 1022 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:46.371165 1022 heartbeater.cc:511] Master 127.0.99.190:44819 requested a full tablet report, sending...
I20240503 01:58:46.372268 441 ts_manager.cc:194] Registered new tserver with Master: 805619f602d2475bb1e6dffa668836ef (127.0.99.132:39835)
I20240503 01:58:46.372853 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:40101
W20240503 01:58:46.484220 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:46.484464 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:46.484539 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:46.487762 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:46.487921 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 01:58:46.493072 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:44819
--never_fsync=true
--heap_profile_path=/tmp/kudu.1025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:46.494184 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:46.495546 1025 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:46.505234 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:46.506063 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:46.506945 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:46.507860 1025 server_base.cc:1034] running on GCE node
I20240503 01:58:46.508410 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:46.509274 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:46.510434 1025 hybrid_clock.cc:648] HybridClock initialized: now 1714701526510420 us; error 35 us; skew 500 ppm
I20240503 01:58:46.512686 1025 webserver.cc:460] Webserver started at http://127.0.99.133:45245/ using document root <none> and password file <none>
I20240503 01:58:46.513382 1025 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:46.513525 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:46.513868 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:46.515662 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "e8b7bc038dd643f081b9d018dd07816d"
format_stamp: "Formatted at 2024-05-03 01:58:46 on dist-test-slave-w9tw"
I20240503 01:58:46.516117 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "e8b7bc038dd643f081b9d018dd07816d"
format_stamp: "Formatted at 2024-05-03 01:58:46 on dist-test-slave-w9tw"
I20240503 01:58:46.519783 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 01:58:46.523177 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:46.524466 1025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 01:58:46.524590 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e8b7bc038dd643f081b9d018dd07816d"
format_stamp: "Formatted at 2024-05-03 01:58:46 on dist-test-slave-w9tw"
I20240503 01:58:46.524703 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:46.541723 1025 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:46.542398 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:46.542603 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:46.543144 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:46.544049 1025 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:46.544095 1025 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:46.544142 1025 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:46.544174 1025 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:46.554131 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:43177
I20240503 01:58:46.554152 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:43177 every 8 connection(s)
I20240503 01:58:46.555184 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 01:58:46.564921 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 1025
I20240503 01:58:46.565040 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 01:58:46.565131 1153 heartbeater.cc:347] Connected to a master server at 127.0.99.190:44819
I20240503 01:58:46.565343 1153 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:46.565855 1153 heartbeater.cc:511] Master 127.0.99.190:44819 requested a full tablet report, sending...
I20240503 01:58:46.566906 441 ts_manager.cc:194] Registered new tserver with Master: e8b7bc038dd643f081b9d018dd07816d (127.0.99.133:43177)
I20240503 01:58:46.567452 441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:42647
I20240503 01:58:46.567792 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 01:58:46.580433 441 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43750:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 01:58:46.600446 694 tablet_service.cc:1467] Processing CreateTablet for tablet 95c2cc34640341f39a3d0fc659179480 (DEFAULT_TABLE table=TestTable [id=9a1e345fae5d4957a5f7ddfc55b92d77]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:46.600446 559 tablet_service.cc:1467] Processing CreateTablet for tablet 95c2cc34640341f39a3d0fc659179480 (DEFAULT_TABLE table=TestTable [id=9a1e345fae5d4957a5f7ddfc55b92d77]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:46.600446 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 95c2cc34640341f39a3d0fc659179480 (DEFAULT_TABLE table=TestTable [id=9a1e345fae5d4957a5f7ddfc55b92d77]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:46.601378 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95c2cc34640341f39a3d0fc659179480. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:46.601382 559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95c2cc34640341f39a3d0fc659179480. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:46.601382 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95c2cc34640341f39a3d0fc659179480. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:46.607897 1171 tablet_bootstrap.cc:492] T 95c2cc34640341f39a3d0fc659179480 P 0618dbed286a4cc98e484b9cca9e7904: Bootstrap starting.
I20240503 01:58:46.609211 1172 tablet_bootstrap.cc:492] T 95c2cc34640341f39a3d0fc659179480 P e49e914485594fb991f0bc3e21c3e91c: Bootstrap starting.
I20240503 01:58:46.610839 1171 tablet_bootstrap.cc:654] T 95c2cc34640341f39a3d0fc659179480 P 0618dbed286a4cc98e484b9cca9e7904: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:46.610952 1173 tablet_bootstrap.cc:492] T 95c2cc34640341f39a3d0fc659179480 P e8b7bc038dd643f081b9d018dd07816d: Bootstrap starting.
I20240503 01:58:46.611064 1172 tablet_bootstrap.cc:654] T 95c2cc34640341f39a3d0fc659179480 P e49e914485594fb991f0bc3e21c3e91c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:46.611598 1171 log.cc:824] T 95c2cc34640341f39a3d0fc659179480 P 0618dbed286a4cc98e484b9cca9e7904: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:46.611828 1172 log.cc:824] T 95c2cc34640341f39a3d0fc659179480 P e49e914485594fb991f0bc3e21c3e91c: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:46.613101 1173 tablet_bootstrap.cc:654] T 95c2cc34640341f39a3d0fc659179480 P e8b7bc038dd643f081b9d018dd07816d: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:46.613425 1172 tablet_bootstrap.cc:492] T 95c2cc34640341f39a3d0fc659179480 P e49e914485594fb991f0bc3e21c3e91c: No bootstrap required, opened a new log
I20240503 01:58:46.613473 1171 tablet_bootstrap.cc:492] T 95c2cc34640341f39a3d0fc659179480 P 0618dbed286a4cc98e484b9cca9e7904: No bootstrap required, opened a new log
I20240503 01:58:46.613603 1172 ts_tablet_manager.cc:1367] T 95c2cc34640341f39a3d0fc659179480 P e49e914485594fb991f0bc3e21c3e91c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 01:58:46.613641 1171 ts_tablet_manager.cc:1367] T 95c2cc34640341f39a3d0fc659179480 P 0618dbed286a4cc98e484b9cca9e7904: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20240503 01:58:46.613889 1173 log.cc:824] T 95c2cc34640341f39a3d0fc659179480 P e8b7bc038dd643f081b9d018dd07816d: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:46.615816 1173 tablet_bootstrap.cc:492] T 95c2cc34640341f39a3d0fc659179480 P e8b7bc038dd643f081b9d018dd07816d: No bootstrap required, opened a new log
I20240503 01:58:46.616014 1173 ts_tablet_manager.cc:1367] T 95c2cc34640341f39a3d0fc659179480 P e8b7bc038dd643f081b9d018dd07816d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 01:58:46.616500 1172 raft_consensus.cc:348] T 95c2cc34640341f39a3d0fc659179480 P e49e914485594fb991f0bc3e21c3e91c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0618dbed286a4cc98e484b9cca9e7904" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40971 } } peers { permanent_uuid: "e8b7bc038dd643f081b9d018dd07816d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43177 } } peers { permanent_uuid: "e49e914485594fb991f0bc3e21c3e91c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37877 } }
I20240503 01:58:46.616729 1172 raft_consensus.cc:374] T 95c2cc34640341f39a3d0fc659179480 P e49e914485594fb991f0bc3e21c3e91c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:46.616777 1172 raft_consensus.cc:729] T 95c2cc34640341f39a3d0fc659179480 P e49e914485594fb991f0bc3e21c3e91c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49e914485594fb991f0bc3e21c3e91c, State: Initialized, Role: FOLLOWER
I20240503 01:58:46.616811 1171 raft_consensus.cc:348] T 95c2cc34640341f39a3d0fc659179480 P 0618dbed286a4cc98e484b9cca9e7904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0618dbed286a4cc98e484b9cca9e7904" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40971 } } peers { permanent_uuid: "e8b7bc038dd643f081b9d018dd07816d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43177 } } peers { permanent_uuid: "e49e914485594fb991f0bc3e21c3e91c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37877 } }
I20240503 01:58:46.617012 1171 raft_consensus.cc:374] T 95c2cc34640341f39a3d0fc659179480 P 0618dbed286a4cc98e484b9cca9e7904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:46.617058 1171 raft_consensus.cc:729] T 95c2cc34640341f39a3d0fc659179480 P 0618dbed286a4cc98e484b9cca9e7904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0618dbed286a4cc98e484b9cca9e7904, State: Initialized, Role: FOLLOWER
I20240503 01:58:46.617187 1172 consensus_queue.cc:260] T 95c2cc34640341f39a3d0fc659179480 P e49e914485594fb991f0bc3e21c3e91c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0618dbed286a4cc98e484b9cca9e7904" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40971 } } peers { permanent_uuid: "e8b7bc038dd643f081b9d018dd07816d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43177 } } peers { permanent_uuid: "e49e914485594fb991f0bc3e21c3e91c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37877 } }
I20240503 01:58:46.617482 1171 consensus_queue.cc:260] T 95c2cc34640341f39a3d0fc659179480 P 0618dbed286a4cc98e484b9cca9e7904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0618dbed286a4cc98e484b9cca9e7904" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40971 } } peers { permanent_uuid: "e8b7bc038dd643f081b9d018dd07816d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43177 } } peers { permanent_uuid: "e49e914485594fb991f0bc3e21c3e91c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37877 } }
I20240503 01:58:46.618170 1172 ts_tablet_manager.cc:1398] T 95c2cc34640341f39a3d0fc659179480 P e49e914485594fb991f0bc3e21c3e91c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 01:58:46.618278 629 heartbeater.cc:503] Master 127.0.99.190:44819 was elected leader, sending a full tablet report...
I20240503 01:58:46.618394 1171 ts_tablet_manager.cc:1398] T 95c2cc34640341f39a3d0fc659179480 P 0618dbed286a4cc98e484b9cca9e7904: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 01:58:46.618793 760 heartbeater.cc:503] Master 127.0.99.190:44819 was elected leader, sending a full tablet report...
I20240503 01:58:46.622359 1173 raft_consensus.cc:348] T 95c2cc34640341f39a3d0fc659179480 P e8b7bc038dd643f081b9d018dd07816d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0618dbed286a4cc98e484b9cca9e7904" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40971 } } peers { permanent_uuid: "e8b7bc038dd643f081b9d018dd07816d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43177 } } peers { permanent_uuid: "e49e914485594fb991f0bc3e21c3e91c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37877 } }
I20240503 01:58:46.622661 1173 raft_consensus.cc:374] T 95c2cc34640341f39a3d0fc659179480 P e8b7bc038dd643f081b9d018dd07816d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:46.622725 1173 raft_consensus.cc:729] T 95c2cc34640341f39a3d0fc659179480 P e8b7bc038dd643f081b9d018dd07816d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8b7bc038dd643f081b9d018dd07816d, State: Initialized, Role: FOLLOWER
I20240503 01:58:46.623309 1173 consensus_queue.cc:260] T 95c2cc34640341f39a3d0fc659179480 P e8b7bc038dd643f081b9d018dd07816d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0618dbed286a4cc98e484b9cca9e7904" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40971 } } peers { permanent_uuid: "e8b7bc038dd643f081b9d018dd07816d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43177 } } peers { permanent_uuid: "e49e914485594fb991f0bc3e21c3e91c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37877 } }
I20240503 01:58:46.624020 1153 heartbeater.cc:503] Master 127.0.99.190:44819 was elected leader, sending a full tablet report...
I20240503 01:58:46.624251 1173 ts_tablet_manager.cc:1398] T 95c2cc34640341f39a3d0fc659179480 P e8b7bc038dd643f081b9d018dd07816d: Time spent starting tablet: real 0.008s user 0.002s sys 0.006s
I20240503 01:58:46.671690 1177 raft_consensus.cc:482] T 95c2cc34640341f39a3d0fc659179480 P e49e914485594fb991f0bc3e21c3e91c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:46.671885 1177 raft_consensus.cc:504] T 95c2cc34640341f39a3d0fc659179480 P e49e914485594fb991f0bc3e21c3e91c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0618dbed286a4cc98e484b9cca9e7904" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40971 } } peers { permanent_uuid: "e8b7bc038dd643f081b9d018dd07816d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43177 } } peers { permanent_uuid: "e49e914485594fb991f0bc3e21c3e91c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37877 } }
I20240503 01:58:46.672993 1177 leader_election.cc:288] T 95c2cc34640341f39a3d0fc659179480 P e49e914485594fb991f0bc3e21c3e91c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0618dbed286a4cc98e484b9cca9e7904 (127.0.99.130:40971), e8b7bc038dd643f081b9d018dd07816d (127.0.99.133:43177)
I20240503 01:58:46.676890 700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c2cc34640341f39a3d0fc659179480" candidate_uuid: "e49e914485594fb991f0bc3e21c3e91c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0618dbed286a4cc98e484b9cca9e7904" is_pre_election: true
I20240503 01:58:46.676977 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c2cc34640341f39a3d0fc659179480" candidate_uuid: "e49e914485594fb991f0bc3e21c3e91c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8b7bc038dd643f081b9d018dd07816d" is_pre_election: true
I20240503 01:58:46.677220 700 raft_consensus.cc:2454] T 95c2cc34640341f39a3d0fc659179480 P 0618dbed286a4cc98e484b9cca9e7904 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e49e914485594fb991f0bc3e21c3e91c in term 0.
I20240503 01:58:46.677273 1107 raft_consensus.cc:2454] T 95c2cc34640341f39a3d0fc659179480 P e8b7bc038dd643f081b9d018dd07816d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e49e914485594fb991f0bc3e21c3e91c in term 0.
I20240503 01:58:46.677712 517 leader_election.cc:302] T 95c2cc34640341f39a3d0fc659179480 P e49e914485594fb991f0bc3e21c3e91c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0618dbed286a4cc98e484b9cca9e7904, e49e914485594fb991f0bc3e21c3e91c; no voters:
I20240503 01:58:46.677987 1177 raft_consensus.cc:2781] T 95c2cc34640341f39a3d0fc659179480 P e49e914485594fb991f0bc3e21c3e91c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:46.678058 1177 raft_consensus.cc:482] T 95c2cc34640341f39a3d0fc659179480 P e49e914485594fb991f0bc3e21c3e91c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:46.678088 1177 raft_consensus.cc:3037] T 95c2cc34640341f39a3d0fc659179480 P e49e914485594fb991f0bc3e21c3e91c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:46.679050 1177 raft_consensus.cc:504] T 95c2cc34640341f39a3d0fc659179480 P e49e914485594fb991f0bc3e21c3e91c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0618dbed286a4cc98e484b9cca9e7904" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40971 } } peers { permanent_uuid: "e8b7bc038dd643f081b9d018dd07816d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43177 } } peers { permanent_uuid: "e49e914485594fb991f0bc3e21c3e91c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37877 } }
I20240503 01:58:46.679417 1177 leader_election.cc:288] T 95c2cc34640341f39a3d0fc659179480 P e49e914485594fb991f0bc3e21c3e91c [CANDIDATE]: Term 1 election: Requested vote from peers 0618dbed286a4cc98e484b9cca9e7904 (127.0.99.130:40971), e8b7bc038dd643f081b9d018dd07816d (127.0.99.133:43177)
I20240503 01:58:46.679807 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c2cc34640341f39a3d0fc659179480" candidate_uuid: "e49e914485594fb991f0bc3e21c3e91c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8b7bc038dd643f081b9d018dd07816d"
I20240503 01:58:46.679845 700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95c2cc34640341f39a3d0fc659179480" candidate_uuid: "e49e914485594fb991f0bc3e21c3e91c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0618dbed286a4cc98e484b9cca9e7904"
I20240503 01:58:46.679931 1107 raft_consensus.cc:3037] T 95c2cc34640341f39a3d0fc659179480 P e8b7bc038dd643f081b9d018dd07816d [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:46.679997 700 raft_consensus.cc:3037] T 95c2cc34640341f39a3d0fc659179480 P 0618dbed286a4cc98e484b9cca9e7904 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:46.680873 1107 raft_consensus.cc:2454] T 95c2cc34640341f39a3d0fc659179480 P e8b7bc038dd643f081b9d018dd07816d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e49e914485594fb991f0bc3e21c3e91c in term 1.
I20240503 01:58:46.680979 700 raft_consensus.cc:2454] T 95c2cc34640341f39a3d0fc659179480 P 0618dbed286a4cc98e484b9cca9e7904 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e49e914485594fb991f0bc3e21c3e91c in term 1.
I20240503 01:58:46.681195 516 leader_election.cc:302] T 95c2cc34640341f39a3d0fc659179480 P e49e914485594fb991f0bc3e21c3e91c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e49e914485594fb991f0bc3e21c3e91c, e8b7bc038dd643f081b9d018dd07816d; no voters:
I20240503 01:58:46.681352 1177 raft_consensus.cc:2781] T 95c2cc34640341f39a3d0fc659179480 P e49e914485594fb991f0bc3e21c3e91c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:46.681596 1177 raft_consensus.cc:686] T 95c2cc34640341f39a3d0fc659179480 P e49e914485594fb991f0bc3e21c3e91c [term 1 LEADER]: Becoming Leader. State: Replica: e49e914485594fb991f0bc3e21c3e91c, State: Running, Role: LEADER
I20240503 01:58:46.681948 1177 consensus_queue.cc:237] T 95c2cc34640341f39a3d0fc659179480 P e49e914485594fb991f0bc3e21c3e91c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0618dbed286a4cc98e484b9cca9e7904" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40971 } } peers { permanent_uuid: "e8b7bc038dd643f081b9d018dd07816d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43177 } } peers { permanent_uuid: "e49e914485594fb991f0bc3e21c3e91c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37877 } }
I20240503 01:58:46.685086 439 catalog_manager.cc:5484] T 95c2cc34640341f39a3d0fc659179480 P e49e914485594fb991f0bc3e21c3e91c reported cstate change: term changed from 0 to 1, leader changed from <none> to e49e914485594fb991f0bc3e21c3e91c (127.0.99.129). New cstate: current_term: 1 leader_uuid: "e49e914485594fb991f0bc3e21c3e91c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0618dbed286a4cc98e484b9cca9e7904" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 40971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8b7bc038dd643f081b9d018dd07816d" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 43177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e49e914485594fb991f0bc3e21c3e91c" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 37877 } health_report { overall_health: HEALTHY } } }
I20240503 01:58:46.695819 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 01:58:46.697553 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e49e914485594fb991f0bc3e21c3e91c to finish bootstrapping
I20240503 01:58:46.703395 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0618dbed286a4cc98e484b9cca9e7904 to finish bootstrapping
W20240503 01:58:46.708446 761 tablet.cc:2369] T 95c2cc34640341f39a3d0fc659179480 P 0618dbed286a4cc98e484b9cca9e7904: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:58:46.708644 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver cccd6ad0cfec4ee8a7c019a220ca91a6 to finish bootstrapping
I20240503 01:58:46.714933 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 805619f602d2475bb1e6dffa668836ef to finish bootstrapping
I20240503 01:58:46.721182 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e8b7bc038dd643f081b9d018dd07816d to finish bootstrapping
W20240503 01:58:46.763952 630 tablet.cc:2369] T 95c2cc34640341f39a3d0fc659179480 P e49e914485594fb991f0bc3e21c3e91c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 01:58:46.806116 1154 tablet.cc:2369] T 95c2cc34640341f39a3d0fc659179480 P e8b7bc038dd643f081b9d018dd07816d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:58:46.930421 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:46.931874 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:46.933404 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:46.940659 559 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:58:46.955351 1087 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:47.048877 694 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:47.048969 956 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:47.049588 825 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:47.049587 559 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:58:47.074406 1087 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:47.100291 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 501
I20240503 01:58:47.109174 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 632
I20240503 01:58:47.115017 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 763
I20240503 01:58:47.121956 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 894
I20240503 01:58:47.128819 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 1025
I20240503 01:58:47.134904 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 410
2024-05-03T01:58:47Z chronyd exiting
[ OK ] AdminCliTest.RebalancerReportOnly (1883 ms)
[----------] 1 test from AdminCliTest (1883 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/0
I20240503 01:58:47.158017 398 test_util.cc:268] Using random seed: 258739453
I20240503 01:58:47.158838 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 01:58:47.158882 398 ts_itest-base.cc:113] --------------
I20240503 01:58:47.158903 398 ts_itest-base.cc:114] 5 tablet servers
I20240503 01:58:47.158921 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 01:58:47.158947 398 ts_itest-base.cc:116] --------------
2024-05-03T01:58:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T01:58:47Z Disabled control of system clock
I20240503 01:58:47.170966 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:34459
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:44839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:34459
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240503 01:58:47.278211 1260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:47.278484 1260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:47.278546 1260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:47.281678 1260 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:58:47.281764 1260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 01:58:47.281787 1260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:47.281807 1260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 01:58:47.281826 1260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 01:58:47.285622 1260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:34459
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:34459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.1260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:47.286656 1260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:47.287781 1260 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:47.295949 1265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:47.296770 1266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:47.297839 1268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:47.298681 1260 server_base.cc:1034] running on GCE node
I20240503 01:58:47.299247 1260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:47.300235 1260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:47.301383 1260 hybrid_clock.cc:648] HybridClock initialized: now 1714701527301357 us; error 61 us; skew 500 ppm
I20240503 01:58:47.305042 1260 webserver.cc:460] Webserver started at http://127.0.99.190:37331/ using document root <none> and password file <none>
I20240503 01:58:47.305655 1260 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:47.305723 1260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:47.305912 1260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:47.307682 1260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4af582eb965b4cec8ea061236593af1f"
format_stamp: "Formatted at 2024-05-03 01:58:47 on dist-test-slave-w9tw"
I20240503 01:58:47.308179 1260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4af582eb965b4cec8ea061236593af1f"
format_stamp: "Formatted at 2024-05-03 01:58:47 on dist-test-slave-w9tw"
I20240503 01:58:47.311712 1260 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 01:58:47.313961 1274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:47.315100 1260 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:58:47.315259 1260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4af582eb965b4cec8ea061236593af1f"
format_stamp: "Formatted at 2024-05-03 01:58:47 on dist-test-slave-w9tw"
I20240503 01:58:47.315361 1260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:47.343139 1260 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:47.343811 1260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:47.343981 1260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:47.352087 1260 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:34459
I20240503 01:58:47.352092 1326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:34459 every 8 connection(s)
I20240503 01:58:47.353111 1260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 01:58:47.355702 1327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:47.356876 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 1260
I20240503 01:58:47.357122 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 01:58:47.362054 1327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4af582eb965b4cec8ea061236593af1f: Bootstrap starting.
I20240503 01:58:47.364524 1327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4af582eb965b4cec8ea061236593af1f: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:47.365422 1327 log.cc:824] T 00000000000000000000000000000000 P 4af582eb965b4cec8ea061236593af1f: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:47.367203 1327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4af582eb965b4cec8ea061236593af1f: No bootstrap required, opened a new log
I20240503 01:58:47.369719 1327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4af582eb965b4cec8ea061236593af1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4af582eb965b4cec8ea061236593af1f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34459 } }
I20240503 01:58:47.369928 1327 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4af582eb965b4cec8ea061236593af1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:47.369963 1327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4af582eb965b4cec8ea061236593af1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af582eb965b4cec8ea061236593af1f, State: Initialized, Role: FOLLOWER
I20240503 01:58:47.370325 1327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4af582eb965b4cec8ea061236593af1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4af582eb965b4cec8ea061236593af1f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34459 } }
I20240503 01:58:47.370427 1327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4af582eb965b4cec8ea061236593af1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:58:47.370471 1327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4af582eb965b4cec8ea061236593af1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:58:47.370589 1327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4af582eb965b4cec8ea061236593af1f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:47.371364 1327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4af582eb965b4cec8ea061236593af1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4af582eb965b4cec8ea061236593af1f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34459 } }
I20240503 01:58:47.371692 1327 leader_election.cc:302] T 00000000000000000000000000000000 P 4af582eb965b4cec8ea061236593af1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4af582eb965b4cec8ea061236593af1f; no voters:
I20240503 01:58:47.371975 1327 leader_election.cc:288] T 00000000000000000000000000000000 P 4af582eb965b4cec8ea061236593af1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:58:47.372123 1332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4af582eb965b4cec8ea061236593af1f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:47.372339 1332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4af582eb965b4cec8ea061236593af1f [term 1 LEADER]: Becoming Leader. State: Replica: 4af582eb965b4cec8ea061236593af1f, State: Running, Role: LEADER
I20240503 01:58:47.372655 1332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4af582eb965b4cec8ea061236593af1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4af582eb965b4cec8ea061236593af1f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34459 } }
I20240503 01:58:47.373135 1327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4af582eb965b4cec8ea061236593af1f [sys.catalog]: configured and running, proceeding with master startup.
I20240503 01:58:47.374276 1333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4af582eb965b4cec8ea061236593af1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4af582eb965b4cec8ea061236593af1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4af582eb965b4cec8ea061236593af1f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34459 } } }
I20240503 01:58:47.374399 1333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4af582eb965b4cec8ea061236593af1f [sys.catalog]: This master's current role is: LEADER
I20240503 01:58:47.374351 1334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4af582eb965b4cec8ea061236593af1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4af582eb965b4cec8ea061236593af1f. Latest consensus state: current_term: 1 leader_uuid: "4af582eb965b4cec8ea061236593af1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4af582eb965b4cec8ea061236593af1f" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 34459 } } }
I20240503 01:58:47.374469 1334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4af582eb965b4cec8ea061236593af1f [sys.catalog]: This master's current role is: LEADER
W20240503 01:58:47.378542 1347 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4af582eb965b4cec8ea061236593af1f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 01:58:47.378640 1347 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 01:58:47.378711 1341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 01:58:47.379431 1341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 01:58:47.384717 1341 catalog_manager.cc:1347] Generated new cluster ID: ec907369bcc64ae899d46951289268fa
I20240503 01:58:47.384809 1341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 01:58:47.399559 1341 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 01:58:47.401333 1341 catalog_manager.cc:1504] Loading token signing keys...
I20240503 01:58:47.409364 1341 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4af582eb965b4cec8ea061236593af1f: Generated new TSK 0
I20240503 01:58:47.410295 1341 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 01:58:47.420884 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34459
--builtin_ntp_servers=127.0.99.148:44839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 01:58:47.572024 1351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:47.572373 1351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:47.572448 1351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:47.576905 1351 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:58:47.577018 1351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:47.577123 1351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 01:58:47.582618 1351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34459
--never_fsync=true
--heap_profile_path=/tmp/kudu.1351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:47.583884 1351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:47.585443 1351 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:47.596504 1356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:47.597365 1357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:47.598206 1359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:47.599205 1351 server_base.cc:1034] running on GCE node
I20240503 01:58:47.599929 1351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:47.600800 1351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:47.602010 1351 hybrid_clock.cc:648] HybridClock initialized: now 1714701527601988 us; error 45 us; skew 500 ppm
I20240503 01:58:47.604236 1351 webserver.cc:460] Webserver started at http://127.0.99.129:37531/ using document root <none> and password file <none>
I20240503 01:58:47.605230 1351 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:47.605347 1351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:47.605589 1351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:47.607409 1351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "388bfcb128e848d0affed9fe9a1d648b"
format_stamp: "Formatted at 2024-05-03 01:58:47 on dist-test-slave-w9tw"
I20240503 01:58:47.607906 1351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "388bfcb128e848d0affed9fe9a1d648b"
format_stamp: "Formatted at 2024-05-03 01:58:47 on dist-test-slave-w9tw"
I20240503 01:58:47.611505 1351 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240503 01:58:47.613867 1365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:47.614979 1351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:47.615099 1351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "388bfcb128e848d0affed9fe9a1d648b"
format_stamp: "Formatted at 2024-05-03 01:58:47 on dist-test-slave-w9tw"
I20240503 01:58:47.615214 1351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:47.629534 1351 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:47.630211 1351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:47.630388 1351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:47.630970 1351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:47.631863 1351 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:47.631913 1351 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:47.631976 1351 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:47.632001 1351 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:47.642488 1351 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:44527
I20240503 01:58:47.642557 1478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:44527 every 8 connection(s)
I20240503 01:58:47.643507 1351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 01:58:47.649081 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 1351
I20240503 01:58:47.649194 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 01:58:47.651719 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34459
--builtin_ntp_servers=127.0.99.148:44839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 01:58:47.653939 1479 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34459
I20240503 01:58:47.654189 1479 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:47.654748 1479 heartbeater.cc:511] Master 127.0.99.190:34459 requested a full tablet report, sending...
I20240503 01:58:47.656152 1291 ts_manager.cc:194] Registered new tserver with Master: 388bfcb128e848d0affed9fe9a1d648b (127.0.99.129:44527)
I20240503 01:58:47.657346 1291 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:43483
W20240503 01:58:47.765565 1482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:47.765836 1482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:47.765915 1482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:47.769954 1482 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:58:47.770066 1482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:47.770184 1482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 01:58:47.774739 1482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34459
--never_fsync=true
--heap_profile_path=/tmp/kudu.1482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:47.775774 1482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:47.776940 1482 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:47.786794 1487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:47.787775 1490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:47.788599 1488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:47.789456 1482 server_base.cc:1034] running on GCE node
I20240503 01:58:47.790167 1482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:47.791054 1482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:47.792286 1482 hybrid_clock.cc:648] HybridClock initialized: now 1714701527792205 us; error 72 us; skew 500 ppm
I20240503 01:58:47.795120 1482 webserver.cc:460] Webserver started at http://127.0.99.130:35177/ using document root <none> and password file <none>
I20240503 01:58:47.795740 1482 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:47.795802 1482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:47.795964 1482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:47.797819 1482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ab28c78a26b8442ead79785775b1ec3a"
format_stamp: "Formatted at 2024-05-03 01:58:47 on dist-test-slave-w9tw"
I20240503 01:58:47.798285 1482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ab28c78a26b8442ead79785775b1ec3a"
format_stamp: "Formatted at 2024-05-03 01:58:47 on dist-test-slave-w9tw"
I20240503 01:58:47.802052 1482 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 01:58:47.804649 1496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:47.805837 1482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:47.805960 1482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ab28c78a26b8442ead79785775b1ec3a"
format_stamp: "Formatted at 2024-05-03 01:58:47 on dist-test-slave-w9tw"
I20240503 01:58:47.806084 1482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:47.833310 1482 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:47.833985 1482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:47.834151 1482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:47.834751 1482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:47.835677 1482 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:47.835726 1482 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:47.835796 1482 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:47.835824 1482 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:47.844226 1482 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:39007
I20240503 01:58:47.844257 1609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:39007 every 8 connection(s)
I20240503 01:58:47.845176 1482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 01:58:47.848955 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 1482
I20240503 01:58:47.849072 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 01:58:47.851394 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34459
--builtin_ntp_servers=127.0.99.148:44839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 01:58:47.855237 1610 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34459
I20240503 01:58:47.855466 1610 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:47.855952 1610 heartbeater.cc:511] Master 127.0.99.190:34459 requested a full tablet report, sending...
I20240503 01:58:47.856921 1291 ts_manager.cc:194] Registered new tserver with Master: ab28c78a26b8442ead79785775b1ec3a (127.0.99.130:39007)
I20240503 01:58:47.857528 1291 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:48099
W20240503 01:58:47.961782 1613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:47.962030 1613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:47.962076 1613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:47.965314 1613 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:58:47.965395 1613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:47.965497 1613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 01:58:47.969269 1613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34459
--never_fsync=true
--heap_profile_path=/tmp/kudu.1613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:47.970239 1613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:47.971441 1613 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:47.980659 1621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:47.981446 1618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:47.982273 1619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:47.983129 1613 server_base.cc:1034] running on GCE node
I20240503 01:58:47.983573 1613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:47.984174 1613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:47.985332 1613 hybrid_clock.cc:648] HybridClock initialized: now 1714701527985313 us; error 34 us; skew 500 ppm
I20240503 01:58:47.987355 1613 webserver.cc:460] Webserver started at http://127.0.99.131:44935/ using document root <none> and password file <none>
I20240503 01:58:47.987905 1613 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:47.987957 1613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:47.988126 1613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:47.989728 1613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "464ac30db1f942bba8fb0165b83ee8a8"
format_stamp: "Formatted at 2024-05-03 01:58:47 on dist-test-slave-w9tw"
I20240503 01:58:47.990196 1613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "464ac30db1f942bba8fb0165b83ee8a8"
format_stamp: "Formatted at 2024-05-03 01:58:47 on dist-test-slave-w9tw"
I20240503 01:58:47.993624 1613 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:58:47.995924 1627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:47.996970 1613 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:47.997073 1613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "464ac30db1f942bba8fb0165b83ee8a8"
format_stamp: "Formatted at 2024-05-03 01:58:47 on dist-test-slave-w9tw"
I20240503 01:58:47.997184 1613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:48.010752 1613 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:48.011348 1613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:48.011510 1613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:48.012018 1613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:48.012849 1613 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:48.012907 1613 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:48.012966 1613 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:48.012995 1613 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:48.021905 1613 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:43007
I20240503 01:58:48.021924 1740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:43007 every 8 connection(s)
I20240503 01:58:48.022835 1613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 01:58:48.027494 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 1613
I20240503 01:58:48.027586 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 01:58:48.029515 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34459
--builtin_ntp_servers=127.0.99.148:44839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 01:58:48.031426 1741 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34459
I20240503 01:58:48.031659 1741 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:48.032088 1741 heartbeater.cc:511] Master 127.0.99.190:34459 requested a full tablet report, sending...
I20240503 01:58:48.033025 1291 ts_manager.cc:194] Registered new tserver with Master: 464ac30db1f942bba8fb0165b83ee8a8 (127.0.99.131:43007)
I20240503 01:58:48.033543 1291 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:37467
W20240503 01:58:48.135691 1744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:48.135931 1744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:48.136006 1744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:48.139082 1744 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:58:48.139160 1744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:48.139258 1744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 01:58:48.142863 1744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34459
--never_fsync=true
--heap_profile_path=/tmp/kudu.1744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:48.143801 1744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:48.144958 1744 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:48.153779 1752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:48.154495 1749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:48.155339 1750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:48.156293 1744 server_base.cc:1034] running on GCE node
I20240503 01:58:48.156764 1744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:48.157354 1744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:48.158576 1744 hybrid_clock.cc:648] HybridClock initialized: now 1714701528158563 us; error 28 us; skew 500 ppm
I20240503 01:58:48.160573 1744 webserver.cc:460] Webserver started at http://127.0.99.132:34653/ using document root <none> and password file <none>
I20240503 01:58:48.161108 1744 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:48.161162 1744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:48.161330 1744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:48.163014 1744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "53cddbda03b1476697db54256013018c"
format_stamp: "Formatted at 2024-05-03 01:58:48 on dist-test-slave-w9tw"
I20240503 01:58:48.163472 1744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "53cddbda03b1476697db54256013018c"
format_stamp: "Formatted at 2024-05-03 01:58:48 on dist-test-slave-w9tw"
I20240503 01:58:48.166865 1744 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240503 01:58:48.169186 1758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:48.170194 1744 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:48.170298 1744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "53cddbda03b1476697db54256013018c"
format_stamp: "Formatted at 2024-05-03 01:58:48 on dist-test-slave-w9tw"
I20240503 01:58:48.170397 1744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:48.218950 1744 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:48.219534 1744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:48.219681 1744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:48.220202 1744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:48.221066 1744 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:48.221112 1744 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:48.221158 1744 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:48.221194 1744 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:48.229682 1744 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:46435
I20240503 01:58:48.229694 1871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:46435 every 8 connection(s)
I20240503 01:58:48.230648 1744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 01:58:48.236006 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 1744
I20240503 01:58:48.236114 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 01:58:48.238355 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34459
--builtin_ntp_servers=127.0.99.148:44839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 01:58:48.239634 1872 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34459
I20240503 01:58:48.239871 1872 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:48.240300 1872 heartbeater.cc:511] Master 127.0.99.190:34459 requested a full tablet report, sending...
I20240503 01:58:48.241189 1291 ts_manager.cc:194] Registered new tserver with Master: 53cddbda03b1476697db54256013018c (127.0.99.132:46435)
I20240503 01:58:48.241727 1291 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:47643
W20240503 01:58:48.348830 1875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:48.349074 1875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:48.349121 1875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:48.352573 1875 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:58:48.352653 1875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:48.352726 1875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 01:58:48.356602 1875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:44839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:34459
--never_fsync=true
--heap_profile_path=/tmp/kudu.1875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:48.357558 1875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:48.358775 1875 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:48.367918 1883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:48.368767 1881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:48.369575 1880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:48.370494 1875 server_base.cc:1034] running on GCE node
I20240503 01:58:48.370954 1875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:48.371562 1875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:48.372807 1875 hybrid_clock.cc:648] HybridClock initialized: now 1714701528372760 us; error 67 us; skew 500 ppm
I20240503 01:58:48.375408 1875 webserver.cc:460] Webserver started at http://127.0.99.133:36197/ using document root <none> and password file <none>
I20240503 01:58:48.376013 1875 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:48.376078 1875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:48.376255 1875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:48.377977 1875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "7a69fddd8c6c44749497d0966f1ce865"
format_stamp: "Formatted at 2024-05-03 01:58:48 on dist-test-slave-w9tw"
I20240503 01:58:48.378458 1875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "7a69fddd8c6c44749497d0966f1ce865"
format_stamp: "Formatted at 2024-05-03 01:58:48 on dist-test-slave-w9tw"
I20240503 01:58:48.381994 1875 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:48.384402 1889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:48.385471 1875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:48.385607 1875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "7a69fddd8c6c44749497d0966f1ce865"
format_stamp: "Formatted at 2024-05-03 01:58:48 on dist-test-slave-w9tw"
I20240503 01:58:48.385707 1875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:48.424651 1875 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:48.425288 1875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:48.425459 1875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:48.426007 1875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:48.426899 1875 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:48.426949 1875 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:48.427011 1875 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:48.427037 1875 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:48.435886 1875 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:36013
I20240503 01:58:48.435915 2002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:36013 every 8 connection(s)
I20240503 01:58:48.436789 1875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 01:58:48.444860 2003 heartbeater.cc:347] Connected to a master server at 127.0.99.190:34459
I20240503 01:58:48.444851 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 1875
I20240503 01:58:48.444967 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 01:58:48.445076 2003 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:48.445559 2003 heartbeater.cc:511] Master 127.0.99.190:34459 requested a full tablet report, sending...
I20240503 01:58:48.446480 1291 ts_manager.cc:194] Registered new tserver with Master: 7a69fddd8c6c44749497d0966f1ce865 (127.0.99.133:36013)
I20240503 01:58:48.447131 1291 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:35453
I20240503 01:58:48.447217 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 01:58:48.456655 1291 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41890:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 01:58:48.474125 1413 tablet_service.cc:1467] Processing CreateTablet for tablet 173dceda8b4d4d6ab465855dc7477acb (DEFAULT_TABLE table=TestTable [id=bd16f31406ad4dfab1af68adbe240397]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:48.474473 1806 tablet_service.cc:1467] Processing CreateTablet for tablet 173dceda8b4d4d6ab465855dc7477acb (DEFAULT_TABLE table=TestTable [id=bd16f31406ad4dfab1af68adbe240397]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:48.474473 1544 tablet_service.cc:1467] Processing CreateTablet for tablet 173dceda8b4d4d6ab465855dc7477acb (DEFAULT_TABLE table=TestTable [id=bd16f31406ad4dfab1af68adbe240397]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:48.475167 1413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 173dceda8b4d4d6ab465855dc7477acb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.475423 1544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 173dceda8b4d4d6ab465855dc7477acb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.475423 1806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 173dceda8b4d4d6ab465855dc7477acb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:48.480252 2021 tablet_bootstrap.cc:492] T 173dceda8b4d4d6ab465855dc7477acb P 388bfcb128e848d0affed9fe9a1d648b: Bootstrap starting.
I20240503 01:58:48.481498 2022 tablet_bootstrap.cc:492] T 173dceda8b4d4d6ab465855dc7477acb P ab28c78a26b8442ead79785775b1ec3a: Bootstrap starting.
I20240503 01:58:48.481719 2023 tablet_bootstrap.cc:492] T 173dceda8b4d4d6ab465855dc7477acb P 53cddbda03b1476697db54256013018c: Bootstrap starting.
I20240503 01:58:48.482919 2021 tablet_bootstrap.cc:654] T 173dceda8b4d4d6ab465855dc7477acb P 388bfcb128e848d0affed9fe9a1d648b: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.483515 2022 tablet_bootstrap.cc:654] T 173dceda8b4d4d6ab465855dc7477acb P ab28c78a26b8442ead79785775b1ec3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.483544 2023 tablet_bootstrap.cc:654] T 173dceda8b4d4d6ab465855dc7477acb P 53cddbda03b1476697db54256013018c: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:48.483757 2021 log.cc:824] T 173dceda8b4d4d6ab465855dc7477acb P 388bfcb128e848d0affed9fe9a1d648b: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:48.484221 2023 log.cc:824] T 173dceda8b4d4d6ab465855dc7477acb P 53cddbda03b1476697db54256013018c: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:48.484236 2022 log.cc:824] T 173dceda8b4d4d6ab465855dc7477acb P ab28c78a26b8442ead79785775b1ec3a: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:48.485528 2021 tablet_bootstrap.cc:492] T 173dceda8b4d4d6ab465855dc7477acb P 388bfcb128e848d0affed9fe9a1d648b: No bootstrap required, opened a new log
I20240503 01:58:48.485708 2021 ts_tablet_manager.cc:1367] T 173dceda8b4d4d6ab465855dc7477acb P 388bfcb128e848d0affed9fe9a1d648b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 01:58:48.485787 2022 tablet_bootstrap.cc:492] T 173dceda8b4d4d6ab465855dc7477acb P ab28c78a26b8442ead79785775b1ec3a: No bootstrap required, opened a new log
I20240503 01:58:48.485787 2023 tablet_bootstrap.cc:492] T 173dceda8b4d4d6ab465855dc7477acb P 53cddbda03b1476697db54256013018c: No bootstrap required, opened a new log
I20240503 01:58:48.485955 2023 ts_tablet_manager.cc:1367] T 173dceda8b4d4d6ab465855dc7477acb P 53cddbda03b1476697db54256013018c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 01:58:48.485955 2022 ts_tablet_manager.cc:1367] T 173dceda8b4d4d6ab465855dc7477acb P ab28c78a26b8442ead79785775b1ec3a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 01:58:48.489578 2021 raft_consensus.cc:348] T 173dceda8b4d4d6ab465855dc7477acb P 388bfcb128e848d0affed9fe9a1d648b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "388bfcb128e848d0affed9fe9a1d648b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44527 } } peers { permanent_uuid: "ab28c78a26b8442ead79785775b1ec3a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39007 } } peers { permanent_uuid: "53cddbda03b1476697db54256013018c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46435 } }
I20240503 01:58:48.489797 2023 raft_consensus.cc:348] T 173dceda8b4d4d6ab465855dc7477acb P 53cddbda03b1476697db54256013018c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "388bfcb128e848d0affed9fe9a1d648b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44527 } } peers { permanent_uuid: "ab28c78a26b8442ead79785775b1ec3a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39007 } } peers { permanent_uuid: "53cddbda03b1476697db54256013018c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46435 } }
I20240503 01:58:48.489800 2022 raft_consensus.cc:348] T 173dceda8b4d4d6ab465855dc7477acb P ab28c78a26b8442ead79785775b1ec3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "388bfcb128e848d0affed9fe9a1d648b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44527 } } peers { permanent_uuid: "ab28c78a26b8442ead79785775b1ec3a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39007 } } peers { permanent_uuid: "53cddbda03b1476697db54256013018c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46435 } }
I20240503 01:58:48.489955 2021 raft_consensus.cc:374] T 173dceda8b4d4d6ab465855dc7477acb P 388bfcb128e848d0affed9fe9a1d648b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.490008 2023 raft_consensus.cc:374] T 173dceda8b4d4d6ab465855dc7477acb P 53cddbda03b1476697db54256013018c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.490041 2022 raft_consensus.cc:374] T 173dceda8b4d4d6ab465855dc7477acb P ab28c78a26b8442ead79785775b1ec3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:48.490062 2023 raft_consensus.cc:729] T 173dceda8b4d4d6ab465855dc7477acb P 53cddbda03b1476697db54256013018c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53cddbda03b1476697db54256013018c, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.490087 2022 raft_consensus.cc:729] T 173dceda8b4d4d6ab465855dc7477acb P ab28c78a26b8442ead79785775b1ec3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab28c78a26b8442ead79785775b1ec3a, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.490093 2021 raft_consensus.cc:729] T 173dceda8b4d4d6ab465855dc7477acb P 388bfcb128e848d0affed9fe9a1d648b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 388bfcb128e848d0affed9fe9a1d648b, State: Initialized, Role: FOLLOWER
I20240503 01:58:48.490553 2022 consensus_queue.cc:260] T 173dceda8b4d4d6ab465855dc7477acb P ab28c78a26b8442ead79785775b1ec3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "388bfcb128e848d0affed9fe9a1d648b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44527 } } peers { permanent_uuid: "ab28c78a26b8442ead79785775b1ec3a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39007 } } peers { permanent_uuid: "53cddbda03b1476697db54256013018c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46435 } }
I20240503 01:58:48.490839 2023 consensus_queue.cc:260] T 173dceda8b4d4d6ab465855dc7477acb P 53cddbda03b1476697db54256013018c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "388bfcb128e848d0affed9fe9a1d648b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44527 } } peers { permanent_uuid: "ab28c78a26b8442ead79785775b1ec3a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39007 } } peers { permanent_uuid: "53cddbda03b1476697db54256013018c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46435 } }
I20240503 01:58:48.491067 2021 consensus_queue.cc:260] T 173dceda8b4d4d6ab465855dc7477acb P 388bfcb128e848d0affed9fe9a1d648b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "388bfcb128e848d0affed9fe9a1d648b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44527 } } peers { permanent_uuid: "ab28c78a26b8442ead79785775b1ec3a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39007 } } peers { permanent_uuid: "53cddbda03b1476697db54256013018c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46435 } }
I20240503 01:58:48.491515 2023 ts_tablet_manager.cc:1398] T 173dceda8b4d4d6ab465855dc7477acb P 53cddbda03b1476697db54256013018c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 01:58:48.491515 2022 ts_tablet_manager.cc:1398] T 173dceda8b4d4d6ab465855dc7477acb P ab28c78a26b8442ead79785775b1ec3a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 01:58:48.491787 1479 heartbeater.cc:503] Master 127.0.99.190:34459 was elected leader, sending a full tablet report...
I20240503 01:58:48.491871 1610 heartbeater.cc:503] Master 127.0.99.190:34459 was elected leader, sending a full tablet report...
I20240503 01:58:48.491886 2021 ts_tablet_manager.cc:1398] T 173dceda8b4d4d6ab465855dc7477acb P 388bfcb128e848d0affed9fe9a1d648b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 01:58:48.492437 1872 heartbeater.cc:503] Master 127.0.99.190:34459 was elected leader, sending a full tablet report...
I20240503 01:58:48.521797 2028 raft_consensus.cc:482] T 173dceda8b4d4d6ab465855dc7477acb P 53cddbda03b1476697db54256013018c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:58:48.521975 2028 raft_consensus.cc:504] T 173dceda8b4d4d6ab465855dc7477acb P 53cddbda03b1476697db54256013018c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "388bfcb128e848d0affed9fe9a1d648b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44527 } } peers { permanent_uuid: "ab28c78a26b8442ead79785775b1ec3a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39007 } } peers { permanent_uuid: "53cddbda03b1476697db54256013018c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46435 } }
I20240503 01:58:48.523018 2028 leader_election.cc:288] T 173dceda8b4d4d6ab465855dc7477acb P 53cddbda03b1476697db54256013018c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 388bfcb128e848d0affed9fe9a1d648b (127.0.99.129:44527), ab28c78a26b8442ead79785775b1ec3a (127.0.99.130:39007)
I20240503 01:58:48.526772 1433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "173dceda8b4d4d6ab465855dc7477acb" candidate_uuid: "53cddbda03b1476697db54256013018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "388bfcb128e848d0affed9fe9a1d648b" is_pre_election: true
I20240503 01:58:48.526860 1564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "173dceda8b4d4d6ab465855dc7477acb" candidate_uuid: "53cddbda03b1476697db54256013018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab28c78a26b8442ead79785775b1ec3a" is_pre_election: true
I20240503 01:58:48.527065 1433 raft_consensus.cc:2454] T 173dceda8b4d4d6ab465855dc7477acb P 388bfcb128e848d0affed9fe9a1d648b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53cddbda03b1476697db54256013018c in term 0.
I20240503 01:58:48.527122 1564 raft_consensus.cc:2454] T 173dceda8b4d4d6ab465855dc7477acb P ab28c78a26b8442ead79785775b1ec3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53cddbda03b1476697db54256013018c in term 0.
I20240503 01:58:48.527493 1760 leader_election.cc:302] T 173dceda8b4d4d6ab465855dc7477acb P 53cddbda03b1476697db54256013018c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 388bfcb128e848d0affed9fe9a1d648b, 53cddbda03b1476697db54256013018c; no voters:
I20240503 01:58:48.527716 2028 raft_consensus.cc:2781] T 173dceda8b4d4d6ab465855dc7477acb P 53cddbda03b1476697db54256013018c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:58:48.527805 2028 raft_consensus.cc:482] T 173dceda8b4d4d6ab465855dc7477acb P 53cddbda03b1476697db54256013018c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:58:48.527856 2028 raft_consensus.cc:3037] T 173dceda8b4d4d6ab465855dc7477acb P 53cddbda03b1476697db54256013018c [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.528714 2028 raft_consensus.cc:504] T 173dceda8b4d4d6ab465855dc7477acb P 53cddbda03b1476697db54256013018c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "388bfcb128e848d0affed9fe9a1d648b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44527 } } peers { permanent_uuid: "ab28c78a26b8442ead79785775b1ec3a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39007 } } peers { permanent_uuid: "53cddbda03b1476697db54256013018c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46435 } }
I20240503 01:58:48.529049 2028 leader_election.cc:288] T 173dceda8b4d4d6ab465855dc7477acb P 53cddbda03b1476697db54256013018c [CANDIDATE]: Term 1 election: Requested vote from peers 388bfcb128e848d0affed9fe9a1d648b (127.0.99.129:44527), ab28c78a26b8442ead79785775b1ec3a (127.0.99.130:39007)
I20240503 01:58:48.529457 1564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "173dceda8b4d4d6ab465855dc7477acb" candidate_uuid: "53cddbda03b1476697db54256013018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab28c78a26b8442ead79785775b1ec3a"
I20240503 01:58:48.529466 1433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "173dceda8b4d4d6ab465855dc7477acb" candidate_uuid: "53cddbda03b1476697db54256013018c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "388bfcb128e848d0affed9fe9a1d648b"
I20240503 01:58:48.529582 1564 raft_consensus.cc:3037] T 173dceda8b4d4d6ab465855dc7477acb P ab28c78a26b8442ead79785775b1ec3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.529582 1433 raft_consensus.cc:3037] T 173dceda8b4d4d6ab465855dc7477acb P 388bfcb128e848d0affed9fe9a1d648b [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:48.530674 1564 raft_consensus.cc:2454] T 173dceda8b4d4d6ab465855dc7477acb P ab28c78a26b8442ead79785775b1ec3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53cddbda03b1476697db54256013018c in term 1.
I20240503 01:58:48.530674 1433 raft_consensus.cc:2454] T 173dceda8b4d4d6ab465855dc7477acb P 388bfcb128e848d0affed9fe9a1d648b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53cddbda03b1476697db54256013018c in term 1.
I20240503 01:58:48.530967 1760 leader_election.cc:302] T 173dceda8b4d4d6ab465855dc7477acb P 53cddbda03b1476697db54256013018c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 388bfcb128e848d0affed9fe9a1d648b, 53cddbda03b1476697db54256013018c; no voters:
I20240503 01:58:48.531147 2028 raft_consensus.cc:2781] T 173dceda8b4d4d6ab465855dc7477acb P 53cddbda03b1476697db54256013018c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:48.531352 2028 raft_consensus.cc:686] T 173dceda8b4d4d6ab465855dc7477acb P 53cddbda03b1476697db54256013018c [term 1 LEADER]: Becoming Leader. State: Replica: 53cddbda03b1476697db54256013018c, State: Running, Role: LEADER
I20240503 01:58:48.531632 2028 consensus_queue.cc:237] T 173dceda8b4d4d6ab465855dc7477acb P 53cddbda03b1476697db54256013018c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "388bfcb128e848d0affed9fe9a1d648b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44527 } } peers { permanent_uuid: "ab28c78a26b8442ead79785775b1ec3a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39007 } } peers { permanent_uuid: "53cddbda03b1476697db54256013018c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46435 } }
I20240503 01:58:48.534456 1289 catalog_manager.cc:5484] T 173dceda8b4d4d6ab465855dc7477acb P 53cddbda03b1476697db54256013018c reported cstate change: term changed from 0 to 1, leader changed from <none> to 53cddbda03b1476697db54256013018c (127.0.99.132). New cstate: current_term: 1 leader_uuid: "53cddbda03b1476697db54256013018c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "388bfcb128e848d0affed9fe9a1d648b" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 44527 } } peers { permanent_uuid: "ab28c78a26b8442ead79785775b1ec3a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39007 } } peers { permanent_uuid: "53cddbda03b1476697db54256013018c" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 46435 } } }
I20240503 01:58:48.552651 398 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 01:58:48.554129 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 388bfcb128e848d0affed9fe9a1d648b to finish bootstrapping
I20240503 01:58:48.559600 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver ab28c78a26b8442ead79785775b1ec3a to finish bootstrapping
I20240503 01:58:48.564594 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 464ac30db1f942bba8fb0165b83ee8a8 to finish bootstrapping
I20240503 01:58:48.570765 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 53cddbda03b1476697db54256013018c to finish bootstrapping
I20240503 01:58:48.576086 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 7a69fddd8c6c44749497d0966f1ce865 to finish bootstrapping
I20240503 01:58:48.582219 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 1351
W20240503 01:58:48.596513 1611 tablet.cc:2369] T 173dceda8b4d4d6ab465855dc7477acb P ab28c78a26b8442ead79785775b1ec3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 01:58:48.731669 1873 tablet.cc:2369] T 173dceda8b4d4d6ab465855dc7477acb P 53cddbda03b1476697db54256013018c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:58:48.771802 1675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:48.772085 1937 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:48.772850 1544 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:48.773069 1806 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:58:48.859027 1675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:48.860206 1806 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:58:48.861001 1937 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:48.867123 1544 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:48.961586 1675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:48.962817 1544 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:48.962857 1937 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:48.963143 1806 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:58:49.035430 1741 heartbeater.cc:503] Master 127.0.99.190:34459 was elected leader, sending a full tablet report...
W20240503 01:58:49.063408 1760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.129:44527: connect: Connection refused (error 111)
W20240503 01:58:49.064391 1760 consensus_peers.cc:485] T 173dceda8b4d4d6ab465855dc7477acb P 53cddbda03b1476697db54256013018c -> Peer 388bfcb128e848d0affed9fe9a1d648b (127.0.99.129:44527): Couldn't send request to peer 388bfcb128e848d0affed9fe9a1d648b. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:44527: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 01:58:49.096046 2124 consensus_queue.cc:1035] T 173dceda8b4d4d6ab465855dc7477acb P 53cddbda03b1476697db54256013018c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab28c78a26b8442ead79785775b1ec3a" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 39007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:58:49.190995 1806 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:58:49.191215 1937 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:49.191802 1544 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:49.192476 1675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:49.265524 1675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:49.265681 1806 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:58:49.266552 1544 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:49.266793 1937 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:49.373693 1675 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:49.374014 1937 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:49.374262 1806 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:58:49.374684 1544 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:49.415817 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 1482
I20240503 01:58:49.422577 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 1613
I20240503 01:58:49.428385 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 1744
I20240503 01:58:49.434293 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 1875
I20240503 01:58:49.439981 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 1260
2024-05-03T01:58:49Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/0 (2304 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2304 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/0
I20240503 01:58:49.462896 398 test_util.cc:268] Using random seed: 261044331
I20240503 01:58:49.463690 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 01:58:49.463728 398 ts_itest-base.cc:113] --------------
I20240503 01:58:49.463748 398 ts_itest-base.cc:114] 7 tablet servers
I20240503 01:58:49.463766 398 ts_itest-base.cc:115] 1 replicas per TS
I20240503 01:58:49.463793 398 ts_itest-base.cc:116] --------------
2024-05-03T01:58:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T01:58:49Z Disabled control of system clock
I20240503 01:58:49.471567 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:35781
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:35781
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240503 01:58:49.580286 2226 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 01:58:49.580540 2226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:49.580567 2226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:49.580641 2226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:49.583734 2226 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:58:49.583813 2226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 01:58:49.583837 2226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:49.583854 2226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 01:58:49.583889 2226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 01:58:49.587662 2226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:35781
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:35781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.2226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:49.588595 2226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:49.589701 2226 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:49.597414 2231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:49.598189 2232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:49.598939 2234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:49.599740 2226 server_base.cc:1034] running on GCE node
I20240503 01:58:49.600234 2226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:49.601131 2226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:49.602300 2226 hybrid_clock.cc:648] HybridClock initialized: now 1714701529602284 us; error 27 us; skew 500 ppm
I20240503 01:58:49.604100 2226 webserver.cc:460] Webserver started at http://127.0.99.190:34309/ using document root <none> and password file <none>
I20240503 01:58:49.604597 2226 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:49.604650 2226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:49.604800 2226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:49.606351 2226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "99706f940467469181bf2d4ab85588bb"
format_stamp: "Formatted at 2024-05-03 01:58:49 on dist-test-slave-w9tw"
I20240503 01:58:49.606853 2226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "99706f940467469181bf2d4ab85588bb"
format_stamp: "Formatted at 2024-05-03 01:58:49 on dist-test-slave-w9tw"
I20240503 01:58:49.610033 2226 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:58:49.612138 2240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:49.613072 2226 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:49.613183 2226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "99706f940467469181bf2d4ab85588bb"
format_stamp: "Formatted at 2024-05-03 01:58:49 on dist-test-slave-w9tw"
I20240503 01:58:49.613278 2226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:49.627298 2226 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:49.627952 2226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:49.628113 2226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:49.634775 2226 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:35781
I20240503 01:58:49.634791 2292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:35781 every 8 connection(s)
I20240503 01:58:49.635743 2226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 01:58:49.636857 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 2226
I20240503 01:58:49.636952 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 01:58:49.638501 2293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:49.643661 2293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99706f940467469181bf2d4ab85588bb: Bootstrap starting.
I20240503 01:58:49.645763 2293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99706f940467469181bf2d4ab85588bb: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:49.646461 2293 log.cc:824] T 00000000000000000000000000000000 P 99706f940467469181bf2d4ab85588bb: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:49.647979 2293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99706f940467469181bf2d4ab85588bb: No bootstrap required, opened a new log
I20240503 01:58:49.650399 2293 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99706f940467469181bf2d4ab85588bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99706f940467469181bf2d4ab85588bb" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35781 } }
I20240503 01:58:49.650612 2293 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99706f940467469181bf2d4ab85588bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:49.650656 2293 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99706f940467469181bf2d4ab85588bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99706f940467469181bf2d4ab85588bb, State: Initialized, Role: FOLLOWER
I20240503 01:58:49.651034 2293 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99706f940467469181bf2d4ab85588bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99706f940467469181bf2d4ab85588bb" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35781 } }
I20240503 01:58:49.651142 2293 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99706f940467469181bf2d4ab85588bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:58:49.651204 2293 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99706f940467469181bf2d4ab85588bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:58:49.651302 2293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99706f940467469181bf2d4ab85588bb [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:49.652014 2293 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99706f940467469181bf2d4ab85588bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99706f940467469181bf2d4ab85588bb" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35781 } }
I20240503 01:58:49.652312 2293 leader_election.cc:302] T 00000000000000000000000000000000 P 99706f940467469181bf2d4ab85588bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99706f940467469181bf2d4ab85588bb; no voters:
I20240503 01:58:49.652566 2293 leader_election.cc:288] T 00000000000000000000000000000000 P 99706f940467469181bf2d4ab85588bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:58:49.652734 2298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99706f940467469181bf2d4ab85588bb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:49.653271 2298 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99706f940467469181bf2d4ab85588bb [term 1 LEADER]: Becoming Leader. State: Replica: 99706f940467469181bf2d4ab85588bb, State: Running, Role: LEADER
I20240503 01:58:49.653568 2298 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99706f940467469181bf2d4ab85588bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99706f940467469181bf2d4ab85588bb" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35781 } }
I20240503 01:58:49.653658 2293 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99706f940467469181bf2d4ab85588bb [sys.catalog]: configured and running, proceeding with master startup.
I20240503 01:58:49.654940 2299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99706f940467469181bf2d4ab85588bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99706f940467469181bf2d4ab85588bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99706f940467469181bf2d4ab85588bb" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35781 } } }
I20240503 01:58:49.655050 2299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99706f940467469181bf2d4ab85588bb [sys.catalog]: This master's current role is: LEADER
I20240503 01:58:49.655336 2300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99706f940467469181bf2d4ab85588bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99706f940467469181bf2d4ab85588bb. Latest consensus state: current_term: 1 leader_uuid: "99706f940467469181bf2d4ab85588bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99706f940467469181bf2d4ab85588bb" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 35781 } } }
I20240503 01:58:49.655448 2300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99706f940467469181bf2d4ab85588bb [sys.catalog]: This master's current role is: LEADER
I20240503 01:58:49.655552 2307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 01:58:49.658087 2307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 01:58:49.662853 2307 catalog_manager.cc:1347] Generated new cluster ID: 3a637ed25f3a4cab9f6014e9e8f6f43a
I20240503 01:58:49.662926 2307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 01:58:49.677340 2307 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 01:58:49.678267 2307 catalog_manager.cc:1504] Loading token signing keys...
I20240503 01:58:49.687055 2307 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 99706f940467469181bf2d4ab85588bb: Generated new TSK 0
I20240503 01:58:49.687752 2307 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 01:58:49.692344 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35781
--builtin_ntp_servers=127.0.99.148:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 01:58:49.807448 2317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:49.807658 2317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:49.807703 2317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:49.810864 2317 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:58:49.810938 2317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:49.811028 2317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 01:58:49.814693 2317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35781
--never_fsync=true
--heap_profile_path=/tmp/kudu.2317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:49.815615 2317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:49.816740 2317 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:49.825227 2323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:49.826033 2325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:49.826865 2322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:49.827692 2317 server_base.cc:1034] running on GCE node
I20240503 01:58:49.828078 2317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:49.828613 2317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:49.829784 2317 hybrid_clock.cc:648] HybridClock initialized: now 1714701529829771 us; error 31 us; skew 500 ppm
I20240503 01:58:49.831674 2317 webserver.cc:460] Webserver started at http://127.0.99.129:44865/ using document root <none> and password file <none>
I20240503 01:58:49.832213 2317 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:49.832295 2317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:49.832468 2317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:49.834084 2317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "80a019d9d1c34ccd9487207f90e68ab0"
format_stamp: "Formatted at 2024-05-03 01:58:49 on dist-test-slave-w9tw"
I20240503 01:58:49.834591 2317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "80a019d9d1c34ccd9487207f90e68ab0"
format_stamp: "Formatted at 2024-05-03 01:58:49 on dist-test-slave-w9tw"
I20240503 01:58:49.838302 2317 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:49.840603 2331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:49.841634 2317 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:49.841758 2317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "80a019d9d1c34ccd9487207f90e68ab0"
format_stamp: "Formatted at 2024-05-03 01:58:49 on dist-test-slave-w9tw"
I20240503 01:58:49.841854 2317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:49.869437 2317 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:49.870095 2317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:49.870267 2317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:49.870860 2317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:49.871740 2317 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:49.871789 2317 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:49.871850 2317 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:49.871879 2317 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:49.880527 2317 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:46305
I20240503 01:58:49.880555 2444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:46305 every 8 connection(s)
I20240503 01:58:49.881491 2317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 01:58:49.888278 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 2317
I20240503 01:58:49.888391 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 01:58:49.889880 2445 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35781
I20240503 01:58:49.890133 2445 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:49.890611 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35781
--builtin_ntp_servers=127.0.99.148:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 01:58:49.890717 2445 heartbeater.cc:511] Master 127.0.99.190:35781 requested a full tablet report, sending...
I20240503 01:58:49.892220 2257 ts_manager.cc:194] Registered new tserver with Master: 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129:46305)
I20240503 01:58:49.893945 2257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:33169
W20240503 01:58:49.997354 2448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:49.997573 2448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:49.997618 2448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:50.000726 2448 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:58:50.000804 2448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:50.000874 2448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 01:58:50.004604 2448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35781
--never_fsync=true
--heap_profile_path=/tmp/kudu.2448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:50.005493 2448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:50.006641 2448 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:50.015470 2453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:50.016299 2454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:50.017129 2456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:50.017962 2448 server_base.cc:1034] running on GCE node
I20240503 01:58:50.018332 2448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:50.018961 2448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:50.020136 2448 hybrid_clock.cc:648] HybridClock initialized: now 1714701530020124 us; error 27 us; skew 500 ppm
I20240503 01:58:50.022145 2448 webserver.cc:460] Webserver started at http://127.0.99.130:43471/ using document root <none> and password file <none>
I20240503 01:58:50.022747 2448 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:50.022805 2448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:50.022980 2448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:50.024616 2448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "86ed23734e434f8784885ebf9f1423b2"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-w9tw"
I20240503 01:58:50.025202 2448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "86ed23734e434f8784885ebf9f1423b2"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-w9tw"
I20240503 01:58:50.029227 2448 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20240503 01:58:50.031580 2462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:50.032650 2448 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:58:50.032766 2448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "86ed23734e434f8784885ebf9f1423b2"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-w9tw"
I20240503 01:58:50.032869 2448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:50.052989 2448 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:50.053725 2448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:50.053886 2448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:50.054469 2448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:50.055397 2448 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:50.055445 2448 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:50.055487 2448 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:50.055509 2448 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:50.063905 2448 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:33335
I20240503 01:58:50.063937 2575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:33335 every 8 connection(s)
I20240503 01:58:50.064859 2448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 01:58:50.067176 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 2448
I20240503 01:58:50.067322 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 01:58:50.069375 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35781
--builtin_ntp_servers=127.0.99.148:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 01:58:50.073940 2576 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35781
I20240503 01:58:50.074205 2576 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:50.074856 2576 heartbeater.cc:511] Master 127.0.99.190:35781 requested a full tablet report, sending...
I20240503 01:58:50.075954 2257 ts_manager.cc:194] Registered new tserver with Master: 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130:33335)
I20240503 01:58:50.076517 2257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:40401
W20240503 01:58:50.180790 2579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:50.181052 2579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:50.181099 2579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:50.184427 2579 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:58:50.184513 2579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:50.184588 2579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 01:58:50.188596 2579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35781
--never_fsync=true
--heap_profile_path=/tmp/kudu.2579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:50.189576 2579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:50.190768 2579 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:50.199738 2584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:50.200531 2585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:50.201341 2587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:50.202148 2579 server_base.cc:1034] running on GCE node
I20240503 01:58:50.202560 2579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:50.203161 2579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:50.204342 2579 hybrid_clock.cc:648] HybridClock initialized: now 1714701530204316 us; error 41 us; skew 500 ppm
I20240503 01:58:50.206362 2579 webserver.cc:460] Webserver started at http://127.0.99.131:36433/ using document root <none> and password file <none>
I20240503 01:58:50.206962 2579 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:50.207022 2579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:50.207201 2579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:50.208894 2579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "cc5fffa9b8844c6f9c948344d8081833"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-w9tw"
I20240503 01:58:50.209403 2579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "cc5fffa9b8844c6f9c948344d8081833"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-w9tw"
I20240503 01:58:50.212821 2579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:58:50.215101 2593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:50.216326 2579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:58:50.216437 2579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "cc5fffa9b8844c6f9c948344d8081833"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-w9tw"
I20240503 01:58:50.216545 2579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:50.229948 2579 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:50.230692 2579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:50.230862 2579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:50.231422 2579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:50.232337 2579 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:50.232385 2579 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:50.232431 2579 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:50.232462 2579 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:50.241487 2579 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:45579
I20240503 01:58:50.241581 2706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:45579 every 8 connection(s)
I20240503 01:58:50.242442 2579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 01:58:50.245254 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 2579
I20240503 01:58:50.245361 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 01:58:50.247789 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35781
--builtin_ntp_servers=127.0.99.148:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 01:58:50.252008 2707 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35781
I20240503 01:58:50.252271 2707 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:50.252743 2707 heartbeater.cc:511] Master 127.0.99.190:35781 requested a full tablet report, sending...
I20240503 01:58:50.253783 2257 ts_manager.cc:194] Registered new tserver with Master: cc5fffa9b8844c6f9c948344d8081833 (127.0.99.131:45579)
I20240503 01:58:50.254709 2257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:41709
W20240503 01:58:50.356236 2710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:50.356482 2710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:50.356526 2710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:50.359637 2710 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:58:50.359709 2710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:50.359778 2710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 01:58:50.363507 2710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35781
--never_fsync=true
--heap_profile_path=/tmp/kudu.2710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:50.364414 2710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:50.365470 2710 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:50.374054 2715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:50.374892 2716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:50.375676 2718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:50.376581 2710 server_base.cc:1034] running on GCE node
I20240503 01:58:50.377010 2710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:50.377657 2710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:50.378808 2710 hybrid_clock.cc:648] HybridClock initialized: now 1714701530378796 us; error 28 us; skew 500 ppm
I20240503 01:58:50.380868 2710 webserver.cc:460] Webserver started at http://127.0.99.132:33027/ using document root <none> and password file <none>
I20240503 01:58:50.381428 2710 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:50.381490 2710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:50.381668 2710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:50.383394 2710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "65d5a1053fb5445486433fe66d82bdbe"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-w9tw"
I20240503 01:58:50.383870 2710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "65d5a1053fb5445486433fe66d82bdbe"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-w9tw"
I20240503 01:58:50.387266 2710 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 01:58:50.389604 2724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:50.390633 2710 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:58:50.390738 2710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "65d5a1053fb5445486433fe66d82bdbe"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-w9tw"
I20240503 01:58:50.390834 2710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:50.405288 2710 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:50.405947 2710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:50.406105 2710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:50.406705 2710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:50.407599 2710 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:50.407650 2710 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:50.407711 2710 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:50.407742 2710 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:50.416492 2710 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:43855
I20240503 01:58:50.416528 2837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:43855 every 8 connection(s)
I20240503 01:58:50.417448 2710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 01:58:50.423939 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 2710
I20240503 01:58:50.424047 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 01:58:50.426132 2838 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35781
I20240503 01:58:50.426267 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35781
--builtin_ntp_servers=127.0.99.148:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 01:58:50.426419 2838 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:50.427008 2838 heartbeater.cc:511] Master 127.0.99.190:35781 requested a full tablet report, sending...
I20240503 01:58:50.428112 2257 ts_manager.cc:194] Registered new tserver with Master: 65d5a1053fb5445486433fe66d82bdbe (127.0.99.132:43855)
I20240503 01:58:50.428762 2257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:49227
W20240503 01:58:50.535732 2841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:50.536008 2841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:50.536088 2841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:50.539834 2841 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:58:50.539938 2841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:50.540045 2841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 01:58:50.544373 2841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35781
--never_fsync=true
--heap_profile_path=/tmp/kudu.2841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:50.545439 2841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:50.546764 2841 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:50.556295 2847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:50.557026 2846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:50.557857 2849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:50.558756 2841 server_base.cc:1034] running on GCE node
I20240503 01:58:50.559196 2841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:50.559811 2841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:50.560988 2841 hybrid_clock.cc:648] HybridClock initialized: now 1714701530560975 us; error 34 us; skew 500 ppm
I20240503 01:58:50.567392 2841 webserver.cc:460] Webserver started at http://127.0.99.133:38549/ using document root <none> and password file <none>
I20240503 01:58:50.568006 2841 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:50.568076 2841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:50.568260 2841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:50.569972 2841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "03ea675b68a74c1192fa1a96f89a494b"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-w9tw"
I20240503 01:58:50.570451 2841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "03ea675b68a74c1192fa1a96f89a494b"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-w9tw"
I20240503 01:58:50.573854 2841 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:50.576104 2855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:50.577219 2841 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:50.577368 2841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "03ea675b68a74c1192fa1a96f89a494b"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-w9tw"
I20240503 01:58:50.577466 2841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:50.613480 2841 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:50.614239 2841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:50.614576 2841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:50.615196 2841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:50.616132 2841 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:50.616191 2841 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:50.616255 2841 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:50.616282 2841 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:50.625701 2841 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:38459
I20240503 01:58:50.625741 2968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:38459 every 8 connection(s)
I20240503 01:58:50.626679 2841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 01:58:50.633251 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 2841
I20240503 01:58:50.633383 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 01:58:50.635459 2969 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35781
I20240503 01:58:50.635493 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35781
--builtin_ntp_servers=127.0.99.148:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 01:58:50.635716 2969 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:50.636292 2969 heartbeater.cc:511] Master 127.0.99.190:35781 requested a full tablet report, sending...
I20240503 01:58:50.637342 2257 ts_manager.cc:194] Registered new tserver with Master: 03ea675b68a74c1192fa1a96f89a494b (127.0.99.133:38459)
I20240503 01:58:50.637912 2257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:54429
W20240503 01:58:50.746670 2972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:50.746915 2972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:50.746985 2972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:50.750142 2972 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:58:50.750242 2972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:50.750340 2972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 01:58:50.754309 2972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35781
--never_fsync=true
--heap_profile_path=/tmp/kudu.2972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:50.755321 2972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:50.756448 2972 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:50.765503 2980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:50.766323 2978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:50.767254 2977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:50.768185 2972 server_base.cc:1034] running on GCE node
I20240503 01:58:50.768656 2972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:50.769281 2972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:50.770455 2972 hybrid_clock.cc:648] HybridClock initialized: now 1714701530770431 us; error 41 us; skew 500 ppm
I20240503 01:58:50.772598 2972 webserver.cc:460] Webserver started at http://127.0.99.134:44977/ using document root <none> and password file <none>
I20240503 01:58:50.773161 2972 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:50.773221 2972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:50.773398 2972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:50.775110 2972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "f0ced467f93d42cdbb6c0a12a812b56a"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-w9tw"
I20240503 01:58:50.775583 2972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "f0ced467f93d42cdbb6c0a12a812b56a"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-w9tw"
I20240503 01:58:50.779026 2972 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 01:58:50.782083 2986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:50.783449 2972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:58:50.783588 2972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "f0ced467f93d42cdbb6c0a12a812b56a"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-w9tw"
I20240503 01:58:50.783702 2972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:50.803262 2972 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:50.803936 2972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:50.804081 2972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:50.804600 2972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:50.805491 2972 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:50.805541 2972 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:50.805583 2972 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:50.805598 2972 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:50.814603 2972 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:33129
I20240503 01:58:50.814621 3099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:33129 every 8 connection(s)
I20240503 01:58:50.815547 2972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 01:58:50.823151 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 2972
I20240503 01:58:50.823292 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 01:58:50.824841 3100 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35781
I20240503 01:58:50.825078 3100 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:50.825439 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35781
--builtin_ntp_servers=127.0.99.148:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 01:58:50.825588 3100 heartbeater.cc:511] Master 127.0.99.190:35781 requested a full tablet report, sending...
I20240503 01:58:50.826772 2257 ts_manager.cc:194] Registered new tserver with Master: f0ced467f93d42cdbb6c0a12a812b56a (127.0.99.134:33129)
I20240503 01:58:50.827353 2257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:48687
I20240503 01:58:50.896368 2445 heartbeater.cc:503] Master 127.0.99.190:35781 was elected leader, sending a full tablet report...
W20240503 01:58:50.936534 3103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:50.936780 3103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:50.936831 3103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:50.940061 3103 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:58:50.940142 3103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:50.940212 3103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 01:58:50.944018 3103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35781
--never_fsync=true
--heap_profile_path=/tmp/kudu.3103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:50.944984 3103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:50.946120 3103 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:50.955962 3109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:50.956732 3108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:50.957551 3111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:50.958567 3103 server_base.cc:1034] running on GCE node
I20240503 01:58:50.958992 3103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:50.959622 3103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:50.960794 3103 hybrid_clock.cc:648] HybridClock initialized: now 1714701530960753 us; error 58 us; skew 500 ppm
I20240503 01:58:50.962882 3103 webserver.cc:460] Webserver started at http://127.0.99.135:42189/ using document root <none> and password file <none>
I20240503 01:58:50.963452 3103 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:50.963532 3103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:50.963714 3103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:58:50.965425 3103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "5372f615183544f8bffc0fe13f4a152f"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-w9tw"
I20240503 01:58:50.965937 3103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "5372f615183544f8bffc0fe13f4a152f"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-w9tw"
I20240503 01:58:50.969336 3103 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240503 01:58:50.971616 3117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:50.972671 3103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:50.972807 3103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "5372f615183544f8bffc0fe13f4a152f"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-w9tw"
I20240503 01:58:50.972919 3103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:50.989425 3103 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:50.990151 3103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:50.990327 3103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:50.990904 3103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:50.991824 3103 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:50.991871 3103 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:50.991915 3103 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:50.991948 3103 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:51.002097 3103 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:45035
I20240503 01:58:51.002187 3230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:45035 every 8 connection(s)
I20240503 01:58:51.003095 3103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 01:58:51.011718 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 3103
I20240503 01:58:51.011775 3231 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35781
I20240503 01:58:51.011848 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 01:58:51.012008 3231 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:51.012480 3231 heartbeater.cc:511] Master 127.0.99.190:35781 requested a full tablet report, sending...
I20240503 01:58:51.013407 2256 ts_manager.cc:194] Registered new tserver with Master: 5372f615183544f8bffc0fe13f4a152f (127.0.99.135:45035)
I20240503 01:58:51.014084 2256 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:53671
I20240503 01:58:51.025933 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 01:58:51.035593 2256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33992:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 01:58:51.051589 3164 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc7ae29d38b4c539498cb946f1c9eaf (DEFAULT_TABLE table=TestTable [id=85646b29369f466b9fc9d710f1578d26]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:51.052423 3164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc7ae29d38b4c539498cb946f1c9eaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:51.057525 3245 tablet_bootstrap.cc:492] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f: Bootstrap starting.
I20240503 01:58:51.059384 3245 tablet_bootstrap.cc:654] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:51.060546 3245 log.cc:824] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:51.062173 3245 tablet_bootstrap.cc:492] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f: No bootstrap required, opened a new log
I20240503 01:58:51.062366 3245 ts_tablet_manager.cc:1367] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 01:58:51.065320 3245 raft_consensus.cc:348] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5372f615183544f8bffc0fe13f4a152f" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45035 } }
I20240503 01:58:51.065492 3245 raft_consensus.cc:374] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:51.065532 3245 raft_consensus.cc:729] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5372f615183544f8bffc0fe13f4a152f, State: Initialized, Role: FOLLOWER
I20240503 01:58:51.065960 3245 consensus_queue.cc:260] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5372f615183544f8bffc0fe13f4a152f" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45035 } }
I20240503 01:58:51.066076 3245 raft_consensus.cc:388] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:58:51.066134 3245 raft_consensus.cc:482] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:58:51.066272 3245 raft_consensus.cc:3037] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:51.067195 3245 raft_consensus.cc:504] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5372f615183544f8bffc0fe13f4a152f" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45035 } }
I20240503 01:58:51.067515 3245 leader_election.cc:302] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5372f615183544f8bffc0fe13f4a152f; no voters:
I20240503 01:58:51.067828 3245 leader_election.cc:288] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:58:51.067937 3247 raft_consensus.cc:2781] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:51.068147 3247 raft_consensus.cc:686] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f [term 1 LEADER]: Becoming Leader. State: Replica: 5372f615183544f8bffc0fe13f4a152f, State: Running, Role: LEADER
I20240503 01:58:51.068455 3231 heartbeater.cc:503] Master 127.0.99.190:35781 was elected leader, sending a full tablet report...
I20240503 01:58:51.068476 3247 consensus_queue.cc:237] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5372f615183544f8bffc0fe13f4a152f" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45035 } }
I20240503 01:58:51.068702 3245 ts_tablet_manager.cc:1398] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240503 01:58:51.071480 2256 catalog_manager.cc:5484] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5372f615183544f8bffc0fe13f4a152f (127.0.99.135). New cstate: current_term: 1 leader_uuid: "5372f615183544f8bffc0fe13f4a152f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5372f615183544f8bffc0fe13f4a152f" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45035 } } }
I20240503 01:58:51.078317 398 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 01:58:51.078872 2576 heartbeater.cc:503] Master 127.0.99.190:35781 was elected leader, sending a full tablet report...
I20240503 01:58:51.079838 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 80a019d9d1c34ccd9487207f90e68ab0 to finish bootstrapping
I20240503 01:58:51.085513 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 86ed23734e434f8784885ebf9f1423b2 to finish bootstrapping
I20240503 01:58:51.091023 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver cc5fffa9b8844c6f9c948344d8081833 to finish bootstrapping
I20240503 01:58:51.096623 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 65d5a1053fb5445486433fe66d82bdbe to finish bootstrapping
I20240503 01:58:51.102219 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 03ea675b68a74c1192fa1a96f89a494b to finish bootstrapping
I20240503 01:58:51.107558 398 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f0ced467f93d42cdbb6c0a12a812b56a to finish bootstrapping
I20240503 01:58:51.113354 398 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 5372f615183544f8bffc0fe13f4a152f to finish bootstrapping
I20240503 01:58:51.118213 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 2579
I20240503 01:58:51.124444 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 2710
I20240503 01:58:51.130295 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 2841
I20240503 01:58:51.135872 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 2972
I20240503 01:58:51.141443 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 3103
I20240503 01:58:54.899261 2256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33992:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:58:54.914860 2379 tablet_service.cc:1467] Processing CreateTablet for tablet 7d729ec44ced4c01a4b84de1fcac3a6f (DEFAULT_TABLE table=rebalance_test_table_0 [id=d658689f115240b580f541db89933aa5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:54.914868 2378 tablet_service.cc:1467] Processing CreateTablet for tablet 32f685df61f144628bbb19c89e36ee6e (DEFAULT_TABLE table=rebalance_test_table_0 [id=d658689f115240b580f541db89933aa5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:54.915063 2510 tablet_service.cc:1467] Processing CreateTablet for tablet 5c289e9ad385459aa89fa32637a42f36 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d658689f115240b580f541db89933aa5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:54.916146 2378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32f685df61f144628bbb19c89e36ee6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:54.916158 2510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c289e9ad385459aa89fa32637a42f36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:54.916698 2379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d729ec44ced4c01a4b84de1fcac3a6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:54.923357 3267 tablet_bootstrap.cc:492] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2: Bootstrap starting.
I20240503 01:58:54.923871 3268 tablet_bootstrap.cc:492] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0: Bootstrap starting.
I20240503 01:58:54.925572 3268 tablet_bootstrap.cc:654] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:54.925590 3267 tablet_bootstrap.cc:654] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:54.926271 3268 log.cc:824] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:54.926271 3267 log.cc:824] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:54.928090 3267 tablet_bootstrap.cc:492] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2: No bootstrap required, opened a new log
I20240503 01:58:54.928099 3268 tablet_bootstrap.cc:492] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0: No bootstrap required, opened a new log
I20240503 01:58:54.928272 3268 ts_tablet_manager.cc:1367] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 01:58:54.928272 3267 ts_tablet_manager.cc:1367] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 01:58:54.931326 3268 raft_consensus.cc:348] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:54.931350 3267 raft_consensus.cc:348] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:58:54.931524 3268 raft_consensus.cc:374] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:54.931525 3267 raft_consensus.cc:374] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:54.931566 3268 raft_consensus.cc:729] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a019d9d1c34ccd9487207f90e68ab0, State: Initialized, Role: FOLLOWER
I20240503 01:58:54.931576 3267 raft_consensus.cc:729] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86ed23734e434f8784885ebf9f1423b2, State: Initialized, Role: FOLLOWER
I20240503 01:58:54.931998 3268 consensus_queue.cc:260] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:54.931998 3267 consensus_queue.cc:260] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:58:54.932121 3268 raft_consensus.cc:388] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:58:54.932125 3267 raft_consensus.cc:388] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:58:54.932181 3268 raft_consensus.cc:482] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:58:54.932185 3267 raft_consensus.cc:482] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:58:54.932286 3267 raft_consensus.cc:3037] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:54.932287 3268 raft_consensus.cc:3037] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:54.933146 3268 raft_consensus.cc:504] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:54.933147 3267 raft_consensus.cc:504] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:58:54.933491 3268 leader_election.cc:302] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80a019d9d1c34ccd9487207f90e68ab0; no voters:
I20240503 01:58:54.933491 3267 leader_election.cc:302] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86ed23734e434f8784885ebf9f1423b2; no voters:
I20240503 01:58:54.933815 3268 leader_election.cc:288] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:58:54.933812 3267 leader_election.cc:288] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:58:54.933907 3271 raft_consensus.cc:2781] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:54.934060 3271 raft_consensus.cc:686] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Becoming Leader. State: Replica: 86ed23734e434f8784885ebf9f1423b2, State: Running, Role: LEADER
I20240503 01:58:54.934363 3271 consensus_queue.cc:237] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:58:54.934595 3273 raft_consensus.cc:2781] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:54.934720 3267 ts_tablet_manager.cc:1398] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20240503 01:58:54.934746 3268 ts_tablet_manager.cc:1398] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240503 01:58:54.934828 3273 raft_consensus.cc:686] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Becoming Leader. State: Replica: 80a019d9d1c34ccd9487207f90e68ab0, State: Running, Role: LEADER
I20240503 01:58:54.935119 3268 tablet_bootstrap.cc:492] T 7d729ec44ced4c01a4b84de1fcac3a6f P 80a019d9d1c34ccd9487207f90e68ab0: Bootstrap starting.
I20240503 01:58:54.935149 3273 consensus_queue.cc:237] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:54.937325 3268 tablet_bootstrap.cc:654] T 7d729ec44ced4c01a4b84de1fcac3a6f P 80a019d9d1c34ccd9487207f90e68ab0: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:54.937244 2257 catalog_manager.cc:5484] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "86ed23734e434f8784885ebf9f1423b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } }
I20240503 01:58:54.937945 2256 catalog_manager.cc:5484] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "80a019d9d1c34ccd9487207f90e68ab0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } }
I20240503 01:58:54.938613 3268 tablet_bootstrap.cc:492] T 7d729ec44ced4c01a4b84de1fcac3a6f P 80a019d9d1c34ccd9487207f90e68ab0: No bootstrap required, opened a new log
I20240503 01:58:54.938747 3268 ts_tablet_manager.cc:1367] T 7d729ec44ced4c01a4b84de1fcac3a6f P 80a019d9d1c34ccd9487207f90e68ab0: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:58:54.939239 3268 raft_consensus.cc:348] T 7d729ec44ced4c01a4b84de1fcac3a6f P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:54.939369 3268 raft_consensus.cc:374] T 7d729ec44ced4c01a4b84de1fcac3a6f P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:54.939448 3268 raft_consensus.cc:729] T 7d729ec44ced4c01a4b84de1fcac3a6f P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a019d9d1c34ccd9487207f90e68ab0, State: Initialized, Role: FOLLOWER
I20240503 01:58:54.939585 3268 consensus_queue.cc:260] T 7d729ec44ced4c01a4b84de1fcac3a6f P 80a019d9d1c34ccd9487207f90e68ab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:54.939669 3268 raft_consensus.cc:388] T 7d729ec44ced4c01a4b84de1fcac3a6f P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:58:54.939728 3268 raft_consensus.cc:482] T 7d729ec44ced4c01a4b84de1fcac3a6f P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:58:54.939779 3268 raft_consensus.cc:3037] T 7d729ec44ced4c01a4b84de1fcac3a6f P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:54.941013 3268 raft_consensus.cc:504] T 7d729ec44ced4c01a4b84de1fcac3a6f P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:54.941179 3268 leader_election.cc:302] T 7d729ec44ced4c01a4b84de1fcac3a6f P 80a019d9d1c34ccd9487207f90e68ab0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80a019d9d1c34ccd9487207f90e68ab0; no voters:
I20240503 01:58:54.941305 3268 leader_election.cc:288] T 7d729ec44ced4c01a4b84de1fcac3a6f P 80a019d9d1c34ccd9487207f90e68ab0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:58:54.941387 3272 raft_consensus.cc:2781] T 7d729ec44ced4c01a4b84de1fcac3a6f P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:54.941555 3268 ts_tablet_manager.cc:1398] T 7d729ec44ced4c01a4b84de1fcac3a6f P 80a019d9d1c34ccd9487207f90e68ab0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:54.941593 3272 raft_consensus.cc:686] T 7d729ec44ced4c01a4b84de1fcac3a6f P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Becoming Leader. State: Replica: 80a019d9d1c34ccd9487207f90e68ab0, State: Running, Role: LEADER
I20240503 01:58:54.941726 3272 consensus_queue.cc:237] T 7d729ec44ced4c01a4b84de1fcac3a6f P 80a019d9d1c34ccd9487207f90e68ab0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:54.943822 2257 catalog_manager.cc:5484] T 7d729ec44ced4c01a4b84de1fcac3a6f P 80a019d9d1c34ccd9487207f90e68ab0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "80a019d9d1c34ccd9487207f90e68ab0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } }
W20240503 01:58:55.070219 3284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:55.070860 3284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:55.075798 3284 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 78.4514 ms
time per row: 0.0392257 ms
I20240503 01:58:55.205513 2256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33992:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:58:55.212424 2379 tablet_service.cc:1467] Processing CreateTablet for tablet ae344bce9aeb452b848accc96ddbb1ac (DEFAULT_TABLE table=rebalance_test_table_1 [id=2ecbe129f8fd43008007db41bf05979c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:55.212427 2510 tablet_service.cc:1467] Processing CreateTablet for tablet a614380f92f64189ac5b5cdb42edf2e6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2ecbe129f8fd43008007db41bf05979c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:55.212622 2509 tablet_service.cc:1467] Processing CreateTablet for tablet 3e1fa0769a934b40a4a4ee536f96e5f4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=2ecbe129f8fd43008007db41bf05979c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:55.212770 2379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae344bce9aeb452b848accc96ddbb1ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:55.212771 2510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a614380f92f64189ac5b5cdb42edf2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:55.213205 2509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e1fa0769a934b40a4a4ee536f96e5f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:55.215256 3268 tablet_bootstrap.cc:492] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0: Bootstrap starting.
I20240503 01:58:55.215672 3267 tablet_bootstrap.cc:492] T 3e1fa0769a934b40a4a4ee536f96e5f4 P 86ed23734e434f8784885ebf9f1423b2: Bootstrap starting.
I20240503 01:58:55.216156 3268 tablet_bootstrap.cc:654] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:55.216554 3267 tablet_bootstrap.cc:654] T 3e1fa0769a934b40a4a4ee536f96e5f4 P 86ed23734e434f8784885ebf9f1423b2: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:55.217222 3268 tablet_bootstrap.cc:492] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0: No bootstrap required, opened a new log
I20240503 01:58:55.217322 3268 ts_tablet_manager.cc:1367] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:55.217690 3267 tablet_bootstrap.cc:492] T 3e1fa0769a934b40a4a4ee536f96e5f4 P 86ed23734e434f8784885ebf9f1423b2: No bootstrap required, opened a new log
I20240503 01:58:55.217792 3267 ts_tablet_manager.cc:1367] T 3e1fa0769a934b40a4a4ee536f96e5f4 P 86ed23734e434f8784885ebf9f1423b2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:55.217744 3268 raft_consensus.cc:348] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:55.217860 3268 raft_consensus.cc:374] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:55.217907 3268 raft_consensus.cc:729] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a019d9d1c34ccd9487207f90e68ab0, State: Initialized, Role: FOLLOWER
I20240503 01:58:55.218039 3268 consensus_queue.cc:260] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:55.218115 3268 raft_consensus.cc:388] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:58:55.218139 3268 raft_consensus.cc:482] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:58:55.218189 3268 raft_consensus.cc:3037] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:55.218361 3267 raft_consensus.cc:348] T 3e1fa0769a934b40a4a4ee536f96e5f4 P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:58:55.218447 3267 raft_consensus.cc:374] T 3e1fa0769a934b40a4a4ee536f96e5f4 P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:55.218482 3267 raft_consensus.cc:729] T 3e1fa0769a934b40a4a4ee536f96e5f4 P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86ed23734e434f8784885ebf9f1423b2, State: Initialized, Role: FOLLOWER
I20240503 01:58:55.218632 3267 consensus_queue.cc:260] T 3e1fa0769a934b40a4a4ee536f96e5f4 P 86ed23734e434f8784885ebf9f1423b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:58:55.218708 3267 raft_consensus.cc:388] T 3e1fa0769a934b40a4a4ee536f96e5f4 P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:58:55.218744 3267 raft_consensus.cc:482] T 3e1fa0769a934b40a4a4ee536f96e5f4 P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:58:55.218798 3267 raft_consensus.cc:3037] T 3e1fa0769a934b40a4a4ee536f96e5f4 P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:55.218956 3268 raft_consensus.cc:504] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:55.219080 3268 leader_election.cc:302] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80a019d9d1c34ccd9487207f90e68ab0; no voters:
I20240503 01:58:55.219195 3268 leader_election.cc:288] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:58:55.219252 3272 raft_consensus.cc:2781] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:55.219347 3272 raft_consensus.cc:686] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Becoming Leader. State: Replica: 80a019d9d1c34ccd9487207f90e68ab0, State: Running, Role: LEADER
I20240503 01:58:55.219455 3268 ts_tablet_manager.cc:1398] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:55.219460 3272 consensus_queue.cc:237] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:55.219679 3267 raft_consensus.cc:504] T 3e1fa0769a934b40a4a4ee536f96e5f4 P 86ed23734e434f8784885ebf9f1423b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:58:55.219800 3267 leader_election.cc:302] T 3e1fa0769a934b40a4a4ee536f96e5f4 P 86ed23734e434f8784885ebf9f1423b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86ed23734e434f8784885ebf9f1423b2; no voters:
I20240503 01:58:55.220220 3271 raft_consensus.cc:2781] T 3e1fa0769a934b40a4a4ee536f96e5f4 P 86ed23734e434f8784885ebf9f1423b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:55.220309 3271 raft_consensus.cc:686] T 3e1fa0769a934b40a4a4ee536f96e5f4 P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Becoming Leader. State: Replica: 86ed23734e434f8784885ebf9f1423b2, State: Running, Role: LEADER
I20240503 01:58:55.220419 3271 consensus_queue.cc:237] T 3e1fa0769a934b40a4a4ee536f96e5f4 P 86ed23734e434f8784885ebf9f1423b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:58:55.221565 3267 leader_election.cc:288] T 3e1fa0769a934b40a4a4ee536f96e5f4 P 86ed23734e434f8784885ebf9f1423b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:58:55.221652 2257 catalog_manager.cc:5484] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "80a019d9d1c34ccd9487207f90e68ab0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } }
I20240503 01:58:55.222182 2256 catalog_manager.cc:5484] T 3e1fa0769a934b40a4a4ee536f96e5f4 P 86ed23734e434f8784885ebf9f1423b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "86ed23734e434f8784885ebf9f1423b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } }
I20240503 01:58:55.222726 3267 ts_tablet_manager.cc:1398] T 3e1fa0769a934b40a4a4ee536f96e5f4 P 86ed23734e434f8784885ebf9f1423b2: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 01:58:55.222903 3267 tablet_bootstrap.cc:492] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2: Bootstrap starting.
I20240503 01:58:55.223845 3267 tablet_bootstrap.cc:654] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:55.225461 3267 tablet_bootstrap.cc:492] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2: No bootstrap required, opened a new log
I20240503 01:58:55.225557 3267 ts_tablet_manager.cc:1367] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:58:55.225946 3267 raft_consensus.cc:348] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:58:55.226037 3267 raft_consensus.cc:374] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:55.226069 3267 raft_consensus.cc:729] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86ed23734e434f8784885ebf9f1423b2, State: Initialized, Role: FOLLOWER
I20240503 01:58:55.226202 3267 consensus_queue.cc:260] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:58:55.226275 3267 raft_consensus.cc:388] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:58:55.226303 3267 raft_consensus.cc:482] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:58:55.226336 3267 raft_consensus.cc:3037] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:55.227483 3267 raft_consensus.cc:504] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:58:55.227598 3267 leader_election.cc:302] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86ed23734e434f8784885ebf9f1423b2; no voters:
I20240503 01:58:55.227692 3267 leader_election.cc:288] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:58:55.227794 3275 raft_consensus.cc:2781] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:55.227890 3275 raft_consensus.cc:686] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Becoming Leader. State: Replica: 86ed23734e434f8784885ebf9f1423b2, State: Running, Role: LEADER
I20240503 01:58:55.227913 3267 ts_tablet_manager.cc:1398] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:55.228029 3275 consensus_queue.cc:237] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:58:55.229518 2257 catalog_manager.cc:5484] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "86ed23734e434f8784885ebf9f1423b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } }
W20240503 01:58:55.357162 3324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:55.357506 3324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:55.360754 3324 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 75.3793 ms
time per row: 0.0376897 ms
I20240503 01:58:55.488180 2255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33992:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:58:55.495641 2509 tablet_service.cc:1467] Processing CreateTablet for tablet 833d9ce859f84b499e667d86965d9fff (DEFAULT_TABLE table=rebalance_test_table_2 [id=5e9078a735bc4854ab7fe5df318eca11]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:55.495637 2379 tablet_service.cc:1467] Processing CreateTablet for tablet 7710730e9d5b440d98bd522c2f808fa3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=5e9078a735bc4854ab7fe5df318eca11]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:55.495637 2378 tablet_service.cc:1467] Processing CreateTablet for tablet 358379b5f00a41d69db831501091e702 (DEFAULT_TABLE table=rebalance_test_table_2 [id=5e9078a735bc4854ab7fe5df318eca11]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:55.496006 2509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 833d9ce859f84b499e667d86965d9fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:55.496007 2379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7710730e9d5b440d98bd522c2f808fa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:55.496202 2378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 358379b5f00a41d69db831501091e702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:55.497941 3268 tablet_bootstrap.cc:492] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0: Bootstrap starting.
I20240503 01:58:55.498199 3267 tablet_bootstrap.cc:492] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2: Bootstrap starting.
I20240503 01:58:55.499006 3268 tablet_bootstrap.cc:654] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:55.499059 3267 tablet_bootstrap.cc:654] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:55.500025 3268 tablet_bootstrap.cc:492] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0: No bootstrap required, opened a new log
I20240503 01:58:55.500027 3267 tablet_bootstrap.cc:492] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2: No bootstrap required, opened a new log
I20240503 01:58:55.500097 3267 ts_tablet_manager.cc:1367] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:55.500108 3268 ts_tablet_manager.cc:1367] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:55.500466 3267 raft_consensus.cc:348] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:58:55.500479 3268 raft_consensus.cc:348] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:55.500557 3267 raft_consensus.cc:374] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:55.500557 3268 raft_consensus.cc:374] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:55.500581 3267 raft_consensus.cc:729] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86ed23734e434f8784885ebf9f1423b2, State: Initialized, Role: FOLLOWER
I20240503 01:58:55.500593 3268 raft_consensus.cc:729] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a019d9d1c34ccd9487207f90e68ab0, State: Initialized, Role: FOLLOWER
I20240503 01:58:55.500689 3267 consensus_queue.cc:260] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:58:55.500778 3267 raft_consensus.cc:388] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:58:55.500734 3268 consensus_queue.cc:260] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:55.500818 3267 raft_consensus.cc:482] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:58:55.500854 3268 raft_consensus.cc:388] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:58:55.500878 3267 raft_consensus.cc:3037] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:55.500901 3268 raft_consensus.cc:482] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:58:55.500955 3268 raft_consensus.cc:3037] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:55.501650 3268 raft_consensus.cc:504] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:55.501667 3267 raft_consensus.cc:504] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:58:55.501796 3268 leader_election.cc:302] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80a019d9d1c34ccd9487207f90e68ab0; no voters:
I20240503 01:58:55.501788 3267 leader_election.cc:302] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86ed23734e434f8784885ebf9f1423b2; no voters:
I20240503 01:58:55.501943 3272 raft_consensus.cc:2781] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:55.501942 3271 raft_consensus.cc:2781] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:55.502013 3271 raft_consensus.cc:686] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Becoming Leader. State: Replica: 86ed23734e434f8784885ebf9f1423b2, State: Running, Role: LEADER
I20240503 01:58:55.502020 3272 raft_consensus.cc:686] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Becoming Leader. State: Replica: 80a019d9d1c34ccd9487207f90e68ab0, State: Running, Role: LEADER
I20240503 01:58:55.502135 3272 consensus_queue.cc:237] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:55.502139 3271 consensus_queue.cc:237] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:58:55.502725 3268 leader_election.cc:288] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:58:55.503086 3268 ts_tablet_manager.cc:1398] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:55.503305 3268 tablet_bootstrap.cc:492] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0: Bootstrap starting.
I20240503 01:58:55.503541 2255 catalog_manager.cc:5484] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "80a019d9d1c34ccd9487207f90e68ab0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } }
I20240503 01:58:55.503541 2256 catalog_manager.cc:5484] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "86ed23734e434f8784885ebf9f1423b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } }
I20240503 01:58:55.504277 3267 leader_election.cc:288] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:58:55.504361 3268 tablet_bootstrap.cc:654] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:55.504521 3267 ts_tablet_manager.cc:1398] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:58:55.505512 3268 tablet_bootstrap.cc:492] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0: No bootstrap required, opened a new log
I20240503 01:58:55.505645 3268 ts_tablet_manager.cc:1367] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:55.506105 3268 raft_consensus.cc:348] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:55.506197 3268 raft_consensus.cc:374] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:55.506232 3268 raft_consensus.cc:729] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a019d9d1c34ccd9487207f90e68ab0, State: Initialized, Role: FOLLOWER
I20240503 01:58:55.506388 3268 consensus_queue.cc:260] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:55.506459 3268 raft_consensus.cc:388] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:58:55.506546 3268 raft_consensus.cc:482] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:58:55.506625 3268 raft_consensus.cc:3037] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:55.508086 3268 raft_consensus.cc:504] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:55.508275 3268 leader_election.cc:302] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80a019d9d1c34ccd9487207f90e68ab0; no voters:
I20240503 01:58:55.508383 3268 leader_election.cc:288] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:58:55.508445 3272 raft_consensus.cc:2781] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:55.508561 3272 raft_consensus.cc:686] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Becoming Leader. State: Replica: 80a019d9d1c34ccd9487207f90e68ab0, State: Running, Role: LEADER
I20240503 01:58:55.508689 3272 consensus_queue.cc:237] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:55.508728 3268 ts_tablet_manager.cc:1398] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 01:58:55.512077 2256 catalog_manager.cc:5484] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "80a019d9d1c34ccd9487207f90e68ab0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } }
W20240503 01:58:55.638125 3359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:55.638713 3359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:55.642158 3359 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 73.3111 ms
time per row: 0.0366556 ms
I20240503 01:58:55.775822 2255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33992:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:58:55.783576 2509 tablet_service.cc:1467] Processing CreateTablet for tablet 986d142f7be641ed9a9c338c41ba430a (DEFAULT_TABLE table=rebalance_test_table_3 [id=cc6ae6794e364638a2b3a21c6e7329ea]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:55.783625 2510 tablet_service.cc:1467] Processing CreateTablet for tablet 5426821cad7e4b7b8689e5a6bfbad7ad (DEFAULT_TABLE table=rebalance_test_table_3 [id=cc6ae6794e364638a2b3a21c6e7329ea]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:55.783962 2509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 986d142f7be641ed9a9c338c41ba430a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:55.784189 2510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5426821cad7e4b7b8689e5a6bfbad7ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:55.784394 2379 tablet_service.cc:1467] Processing CreateTablet for tablet 49706355c8864e638e187eee7711cac3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=cc6ae6794e364638a2b3a21c6e7329ea]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:55.784722 2379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49706355c8864e638e187eee7711cac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:55.786260 3267 tablet_bootstrap.cc:492] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2: Bootstrap starting.
I20240503 01:58:55.787397 3268 tablet_bootstrap.cc:492] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0: Bootstrap starting.
I20240503 01:58:55.788267 3267 tablet_bootstrap.cc:654] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:55.788712 3268 tablet_bootstrap.cc:654] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:55.789685 3267 tablet_bootstrap.cc:492] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2: No bootstrap required, opened a new log
I20240503 01:58:55.789793 3267 ts_tablet_manager.cc:1367] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 01:58:55.789938 3268 tablet_bootstrap.cc:492] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0: No bootstrap required, opened a new log
I20240503 01:58:55.790035 3268 ts_tablet_manager.cc:1367] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:55.790455 3267 raft_consensus.cc:348] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:58:55.790663 3267 raft_consensus.cc:374] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:55.790719 3267 raft_consensus.cc:729] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86ed23734e434f8784885ebf9f1423b2, State: Initialized, Role: FOLLOWER
I20240503 01:58:55.790634 3268 raft_consensus.cc:348] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:55.790771 3268 raft_consensus.cc:374] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:55.790818 3268 raft_consensus.cc:729] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a019d9d1c34ccd9487207f90e68ab0, State: Initialized, Role: FOLLOWER
I20240503 01:58:55.790905 3267 consensus_queue.cc:260] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:58:55.790977 3268 consensus_queue.cc:260] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:55.791047 3267 raft_consensus.cc:388] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:58:55.791060 3268 raft_consensus.cc:388] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:58:55.791095 3268 raft_consensus.cc:482] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:58:55.791093 3267 raft_consensus.cc:482] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:58:55.791139 3268 raft_consensus.cc:3037] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:55.791157 3267 raft_consensus.cc:3037] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:55.792021 3267 raft_consensus.cc:504] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:58:55.792155 3267 leader_election.cc:302] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86ed23734e434f8784885ebf9f1423b2; no voters:
I20240503 01:58:55.792155 3268 raft_consensus.cc:504] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:55.792263 3267 leader_election.cc:288] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:58:55.792344 3271 raft_consensus.cc:2781] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:55.792513 3271 raft_consensus.cc:686] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Becoming Leader. State: Replica: 86ed23734e434f8784885ebf9f1423b2, State: Running, Role: LEADER
I20240503 01:58:55.792665 3271 consensus_queue.cc:237] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:58:55.792765 3267 ts_tablet_manager.cc:1398] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 01:58:55.792984 3267 tablet_bootstrap.cc:492] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2: Bootstrap starting.
I20240503 01:58:55.793188 3268 leader_election.cc:302] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80a019d9d1c34ccd9487207f90e68ab0; no voters:
I20240503 01:58:55.793370 3276 raft_consensus.cc:2781] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:55.793466 3276 raft_consensus.cc:686] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Becoming Leader. State: Replica: 80a019d9d1c34ccd9487207f90e68ab0, State: Running, Role: LEADER
I20240503 01:58:55.793607 3276 consensus_queue.cc:237] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:55.794266 3268 leader_election.cc:288] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:58:55.794775 3268 ts_tablet_manager.cc:1398] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 01:58:55.795292 2255 catalog_manager.cc:5484] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "86ed23734e434f8784885ebf9f1423b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } }
I20240503 01:58:55.795326 2256 catalog_manager.cc:5484] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "80a019d9d1c34ccd9487207f90e68ab0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } }
I20240503 01:58:55.796886 3267 tablet_bootstrap.cc:654] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:55.799638 3267 tablet_bootstrap.cc:492] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2: No bootstrap required, opened a new log
I20240503 01:58:55.799785 3267 ts_tablet_manager.cc:1367] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:58:55.800390 3267 raft_consensus.cc:348] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:58:55.800487 3267 raft_consensus.cc:374] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:55.800509 3267 raft_consensus.cc:729] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86ed23734e434f8784885ebf9f1423b2, State: Initialized, Role: FOLLOWER
I20240503 01:58:55.800628 3267 consensus_queue.cc:260] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:58:55.800698 3267 raft_consensus.cc:388] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:58:55.800724 3267 raft_consensus.cc:482] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:58:55.800783 3267 raft_consensus.cc:3037] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:55.801713 3267 raft_consensus.cc:504] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:58:55.801870 3267 leader_election.cc:302] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86ed23734e434f8784885ebf9f1423b2; no voters:
I20240503 01:58:55.801978 3267 leader_election.cc:288] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:58:55.802033 3271 raft_consensus.cc:2781] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:55.802229 3271 raft_consensus.cc:686] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Becoming Leader. State: Replica: 86ed23734e434f8784885ebf9f1423b2, State: Running, Role: LEADER
I20240503 01:58:55.802361 3271 consensus_queue.cc:237] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:58:55.802491 3267 ts_tablet_manager.cc:1398] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 01:58:55.803902 2255 catalog_manager.cc:5484] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "86ed23734e434f8784885ebf9f1423b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } }
W20240503 01:58:55.935662 3392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:55.935971 3392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:55.939313 3392 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 71.2768 ms
time per row: 0.0356384 ms
I20240503 01:58:56.062278 2255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33992:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:58:56.069427 2510 tablet_service.cc:1467] Processing CreateTablet for tablet 5f821e4f2b37483abda9bd5a89cd0b19 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e3217159b04d4bebadaaac3cc7d7cec1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:56.069427 2378 tablet_service.cc:1467] Processing CreateTablet for tablet f586b7fd77824920b4300f38a29ba87b (DEFAULT_TABLE table=rebalance_test_table_4 [id=e3217159b04d4bebadaaac3cc7d7cec1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:56.069447 2379 tablet_service.cc:1467] Processing CreateTablet for tablet c0bce60681a34a96a83284c762d6feee (DEFAULT_TABLE table=rebalance_test_table_4 [id=e3217159b04d4bebadaaac3cc7d7cec1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:58:56.069777 2378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f586b7fd77824920b4300f38a29ba87b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:56.069775 2510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f821e4f2b37483abda9bd5a89cd0b19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:56.069998 2379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0bce60681a34a96a83284c762d6feee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:56.072142 3267 tablet_bootstrap.cc:492] T 5f821e4f2b37483abda9bd5a89cd0b19 P 86ed23734e434f8784885ebf9f1423b2: Bootstrap starting.
I20240503 01:58:56.072418 3268 tablet_bootstrap.cc:492] T f586b7fd77824920b4300f38a29ba87b P 80a019d9d1c34ccd9487207f90e68ab0: Bootstrap starting.
I20240503 01:58:56.073119 3267 tablet_bootstrap.cc:654] T 5f821e4f2b37483abda9bd5a89cd0b19 P 86ed23734e434f8784885ebf9f1423b2: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:56.073210 3268 tablet_bootstrap.cc:654] T f586b7fd77824920b4300f38a29ba87b P 80a019d9d1c34ccd9487207f90e68ab0: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:56.074299 3267 tablet_bootstrap.cc:492] T 5f821e4f2b37483abda9bd5a89cd0b19 P 86ed23734e434f8784885ebf9f1423b2: No bootstrap required, opened a new log
I20240503 01:58:56.074317 3268 tablet_bootstrap.cc:492] T f586b7fd77824920b4300f38a29ba87b P 80a019d9d1c34ccd9487207f90e68ab0: No bootstrap required, opened a new log
I20240503 01:58:56.074388 3267 ts_tablet_manager.cc:1367] T 5f821e4f2b37483abda9bd5a89cd0b19 P 86ed23734e434f8784885ebf9f1423b2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:58:56.074393 3268 ts_tablet_manager.cc:1367] T f586b7fd77824920b4300f38a29ba87b P 80a019d9d1c34ccd9487207f90e68ab0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:58:56.074867 3267 raft_consensus.cc:348] T 5f821e4f2b37483abda9bd5a89cd0b19 P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:58:56.074906 3268 raft_consensus.cc:348] T f586b7fd77824920b4300f38a29ba87b P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:56.074982 3267 raft_consensus.cc:374] T 5f821e4f2b37483abda9bd5a89cd0b19 P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:56.074998 3268 raft_consensus.cc:374] T f586b7fd77824920b4300f38a29ba87b P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:56.075027 3267 raft_consensus.cc:729] T 5f821e4f2b37483abda9bd5a89cd0b19 P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86ed23734e434f8784885ebf9f1423b2, State: Initialized, Role: FOLLOWER
I20240503 01:58:56.075033 3268 raft_consensus.cc:729] T f586b7fd77824920b4300f38a29ba87b P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a019d9d1c34ccd9487207f90e68ab0, State: Initialized, Role: FOLLOWER
I20240503 01:58:56.075155 3268 consensus_queue.cc:260] T f586b7fd77824920b4300f38a29ba87b P 80a019d9d1c34ccd9487207f90e68ab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:56.075166 3267 consensus_queue.cc:260] T 5f821e4f2b37483abda9bd5a89cd0b19 P 86ed23734e434f8784885ebf9f1423b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:58:56.075220 3268 raft_consensus.cc:388] T f586b7fd77824920b4300f38a29ba87b P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:58:56.075253 3268 raft_consensus.cc:482] T f586b7fd77824920b4300f38a29ba87b P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:58:56.075253 3267 raft_consensus.cc:388] T 5f821e4f2b37483abda9bd5a89cd0b19 P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:58:56.075280 3267 raft_consensus.cc:482] T 5f821e4f2b37483abda9bd5a89cd0b19 P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:58:56.075287 3268 raft_consensus.cc:3037] T f586b7fd77824920b4300f38a29ba87b P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:56.075309 3267 raft_consensus.cc:3037] T 5f821e4f2b37483abda9bd5a89cd0b19 P 86ed23734e434f8784885ebf9f1423b2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:56.075944 3267 raft_consensus.cc:504] T 5f821e4f2b37483abda9bd5a89cd0b19 P 86ed23734e434f8784885ebf9f1423b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:58:56.076041 3267 leader_election.cc:302] T 5f821e4f2b37483abda9bd5a89cd0b19 P 86ed23734e434f8784885ebf9f1423b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86ed23734e434f8784885ebf9f1423b2; no voters:
I20240503 01:58:56.076115 3267 leader_election.cc:288] T 5f821e4f2b37483abda9bd5a89cd0b19 P 86ed23734e434f8784885ebf9f1423b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:58:56.076092 3268 raft_consensus.cc:504] T f586b7fd77824920b4300f38a29ba87b P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:56.076200 3268 leader_election.cc:302] T f586b7fd77824920b4300f38a29ba87b P 80a019d9d1c34ccd9487207f90e68ab0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80a019d9d1c34ccd9487207f90e68ab0; no voters:
I20240503 01:58:56.076196 3390 raft_consensus.cc:2781] T 5f821e4f2b37483abda9bd5a89cd0b19 P 86ed23734e434f8784885ebf9f1423b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:56.076277 3268 leader_election.cc:288] T f586b7fd77824920b4300f38a29ba87b P 80a019d9d1c34ccd9487207f90e68ab0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:58:56.076331 3276 raft_consensus.cc:2781] T f586b7fd77824920b4300f38a29ba87b P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:56.076508 3276 raft_consensus.cc:686] T f586b7fd77824920b4300f38a29ba87b P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Becoming Leader. State: Replica: 80a019d9d1c34ccd9487207f90e68ab0, State: Running, Role: LEADER
I20240503 01:58:56.076612 3268 ts_tablet_manager.cc:1398] T f586b7fd77824920b4300f38a29ba87b P 80a019d9d1c34ccd9487207f90e68ab0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:58:56.076629 3276 consensus_queue.cc:237] T f586b7fd77824920b4300f38a29ba87b P 80a019d9d1c34ccd9487207f90e68ab0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:56.076746 3268 tablet_bootstrap.cc:492] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0: Bootstrap starting.
I20240503 01:58:56.077731 3268 tablet_bootstrap.cc:654] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0: Neither blocks nor log segments found. Creating new log.
I20240503 01:58:56.078008 3390 raft_consensus.cc:686] T 5f821e4f2b37483abda9bd5a89cd0b19 P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Becoming Leader. State: Replica: 86ed23734e434f8784885ebf9f1423b2, State: Running, Role: LEADER
I20240503 01:58:56.078155 3390 consensus_queue.cc:237] T 5f821e4f2b37483abda9bd5a89cd0b19 P 86ed23734e434f8784885ebf9f1423b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:58:56.078337 2255 catalog_manager.cc:5484] T f586b7fd77824920b4300f38a29ba87b P 80a019d9d1c34ccd9487207f90e68ab0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "80a019d9d1c34ccd9487207f90e68ab0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } }
I20240503 01:58:56.079471 3267 ts_tablet_manager.cc:1398] T 5f821e4f2b37483abda9bd5a89cd0b19 P 86ed23734e434f8784885ebf9f1423b2: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 01:58:56.079880 2252 catalog_manager.cc:5484] T 5f821e4f2b37483abda9bd5a89cd0b19 P 86ed23734e434f8784885ebf9f1423b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130). New cstate: current_term: 1 leader_uuid: "86ed23734e434f8784885ebf9f1423b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } }
I20240503 01:58:56.083285 3268 tablet_bootstrap.cc:492] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0: No bootstrap required, opened a new log
I20240503 01:58:56.083388 3268 ts_tablet_manager.cc:1367] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240503 01:58:56.083890 3268 raft_consensus.cc:348] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:56.083997 3268 raft_consensus.cc:374] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:58:56.084038 3268 raft_consensus.cc:729] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a019d9d1c34ccd9487207f90e68ab0, State: Initialized, Role: FOLLOWER
I20240503 01:58:56.084666 3268 consensus_queue.cc:260] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:56.084805 3268 raft_consensus.cc:388] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:58:56.084883 3268 raft_consensus.cc:482] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:58:56.084954 3268 raft_consensus.cc:3037] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:58:56.086659 3268 raft_consensus.cc:504] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:56.086881 3268 leader_election.cc:302] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80a019d9d1c34ccd9487207f90e68ab0; no voters:
I20240503 01:58:56.087123 3268 leader_election.cc:288] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:58:56.087203 3420 raft_consensus.cc:2781] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:58:56.087436 3420 raft_consensus.cc:686] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Becoming Leader. State: Replica: 80a019d9d1c34ccd9487207f90e68ab0, State: Running, Role: LEADER
I20240503 01:58:56.087461 3268 ts_tablet_manager.cc:1398] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240503 01:58:56.087564 3420 consensus_queue.cc:237] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:56.089152 2252 catalog_manager.cc:5484] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "80a019d9d1c34ccd9487207f90e68ab0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } }
W20240503 01:58:56.218320 3425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:56.218663 3425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:56.221953 3425 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 101.843 ms
time per row: 0.0509213 ms
I20240503 01:58:56.379019 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:45579
--local_ip_for_outbound_sockets=127.0.99.131
--tserver_master_addrs=127.0.99.190:35781
--webserver_port=36433
--webserver_interface=127.0.99.131
--builtin_ntp_servers=127.0.99.148:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 01:58:56.502882 3452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:56.503127 3452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:56.503173 3452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:56.506440 3452 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:58:56.506554 3452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:56.506656 3452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 01:58:56.510468 3452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:45579
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=36433
--tserver_master_addrs=127.0.99.190:35781
--never_fsync=true
--heap_profile_path=/tmp/kudu.3452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:56.511508 3452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:56.512748 3452 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:56.522820 3457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:56.524173 3460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:56.525067 3458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:56.526224 3452 server_base.cc:1034] running on GCE node
I20240503 01:58:56.526747 3452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:56.527396 3452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:56.528585 3452 hybrid_clock.cc:648] HybridClock initialized: now 1714701536528560 us; error 45 us; skew 500 ppm
I20240503 01:58:56.530813 3452 webserver.cc:460] Webserver started at http://127.0.99.131:36433/ using document root <none> and password file <none>
I20240503 01:58:56.531428 3452 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:56.531512 3452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:56.535851 3452 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 01:58:56.538406 3466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:56.539726 3452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:58:56.539871 3452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "cc5fffa9b8844c6f9c948344d8081833"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-w9tw"
I20240503 01:58:56.540370 3452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:56.560092 3452 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:56.560787 3452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:56.560968 3452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:56.561647 3452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:56.562657 3452 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:56.562750 3452 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:56.562801 3452 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:56.562817 3452 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:56.572265 3452 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:45579
I20240503 01:58:56.572453 3579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:45579 every 8 connection(s)
I20240503 01:58:56.573432 3452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 01:58:56.576795 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 3452
I20240503 01:58:56.577282 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:43855
--local_ip_for_outbound_sockets=127.0.99.132
--tserver_master_addrs=127.0.99.190:35781
--webserver_port=33027
--webserver_interface=127.0.99.132
--builtin_ntp_servers=127.0.99.148:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 01:58:56.584546 3580 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35781
I20240503 01:58:56.584894 3580 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:56.585592 3580 heartbeater.cc:511] Master 127.0.99.190:35781 requested a full tablet report, sending...
I20240503 01:58:56.587266 2253 ts_manager.cc:194] Re-registered known tserver with Master: cc5fffa9b8844c6f9c948344d8081833 (127.0.99.131:45579)
I20240503 01:58:56.588073 2253 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:46219
W20240503 01:58:56.704125 3583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:56.704418 3583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:56.704463 3583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:56.707739 3583 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:58:56.707835 3583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:56.707909 3583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 01:58:56.711916 3583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:43855
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=33027
--tserver_master_addrs=127.0.99.190:35781
--never_fsync=true
--heap_profile_path=/tmp/kudu.3583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:56.712975 3583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:56.714221 3583 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:56.724272 3591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:56.725484 3588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:56.726550 3589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:56.728145 3583 server_base.cc:1034] running on GCE node
I20240503 01:58:56.728693 3583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:56.729351 3583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:56.730573 3583 hybrid_clock.cc:648] HybridClock initialized: now 1714701536730559 us; error 29 us; skew 500 ppm
I20240503 01:58:56.732843 3583 webserver.cc:460] Webserver started at http://127.0.99.132:33027/ using document root <none> and password file <none>
I20240503 01:58:56.733470 3583 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:56.733553 3583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:56.737751 3583 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:56.740726 3597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:56.742264 3583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 01:58:56.742468 3583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "65d5a1053fb5445486433fe66d82bdbe"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-w9tw"
I20240503 01:58:56.743023 3583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:56.785720 3583 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:56.786998 3583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:56.787256 3583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:56.788153 3583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:56.789448 3583 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:56.789515 3583 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:56.789608 3583 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:56.789640 3583 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:56.800384 3583 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:43855
I20240503 01:58:56.800432 3710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:43855 every 8 connection(s)
I20240503 01:58:56.801609 3583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 01:58:56.805481 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 3583
I20240503 01:58:56.805980 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:38459
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:35781
--webserver_port=38549
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 01:58:56.814211 3711 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35781
I20240503 01:58:56.814815 3711 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:56.815703 3711 heartbeater.cc:511] Master 127.0.99.190:35781 requested a full tablet report, sending...
I20240503 01:58:56.817170 2253 ts_manager.cc:194] Re-registered known tserver with Master: 65d5a1053fb5445486433fe66d82bdbe (127.0.99.132:43855)
I20240503 01:58:56.818116 2253 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:50319
W20240503 01:58:56.937564 3714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:56.937827 3714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:56.937877 3714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:56.941172 3714 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:58:56.941269 3714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:56.941341 3714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 01:58:56.945575 3714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:38459
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=38549
--tserver_master_addrs=127.0.99.190:35781
--never_fsync=true
--heap_profile_path=/tmp/kudu.3714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:56.946615 3714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:56.947863 3714 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:56.957986 3719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:56.958856 3722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:56.959843 3720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:56.961138 3714 server_base.cc:1034] running on GCE node
I20240503 01:58:56.961642 3714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:56.962280 3714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:56.963537 3714 hybrid_clock.cc:648] HybridClock initialized: now 1714701536963509 us; error 55 us; skew 500 ppm
I20240503 01:58:56.965724 3714 webserver.cc:460] Webserver started at http://127.0.99.133:38549/ using document root <none> and password file <none>
I20240503 01:58:56.966318 3714 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:56.966401 3714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:56.970662 3714 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:58:56.973531 3728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:56.975164 3714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:58:56.975324 3714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "03ea675b68a74c1192fa1a96f89a494b"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-w9tw"
I20240503 01:58:56.975997 3714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:56.990576 3714 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:56.991488 3714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:56.991698 3714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:56.992378 3714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:56.993465 3714 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:56.993516 3714 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:56.993579 3714 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:56.993610 3714 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:57.005053 3714 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:38459
I20240503 01:58:57.005095 3841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:38459 every 8 connection(s)
I20240503 01:58:57.006170 3714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 01:58:57.014588 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 3714
I20240503 01:58:57.015012 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:33129
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:35781
--webserver_port=44977
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 01:58:57.015925 3842 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35781
I20240503 01:58:57.016216 3842 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:57.016955 3842 heartbeater.cc:511] Master 127.0.99.190:35781 requested a full tablet report, sending...
I20240503 01:58:57.018165 2253 ts_manager.cc:194] Re-registered known tserver with Master: 03ea675b68a74c1192fa1a96f89a494b (127.0.99.133:38459)
I20240503 01:58:57.018782 2253 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:46355
W20240503 01:58:57.155314 3845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:57.155596 3845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:57.155668 3845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:57.159066 3845 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:58:57.159166 3845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:57.159274 3845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 01:58:57.163616 3845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:33129
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=44977
--tserver_master_addrs=127.0.99.190:35781
--never_fsync=true
--heap_profile_path=/tmp/kudu.3845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:57.164716 3845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:57.166005 3845 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:57.178458 3850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:57.179378 3851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:57.180564 3853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:57.181970 3845 server_base.cc:1034] running on GCE node
I20240503 01:58:57.182490 3845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:57.183315 3845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:57.184545 3845 hybrid_clock.cc:648] HybridClock initialized: now 1714701537184495 us; error 73 us; skew 500 ppm
I20240503 01:58:57.187065 3845 webserver.cc:460] Webserver started at http://127.0.99.134:44977/ using document root <none> and password file <none>
I20240503 01:58:57.187705 3845 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:57.187790 3845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:57.192412 3845 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 01:58:57.194860 3859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:57.196256 3845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:58:57.196405 3845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "f0ced467f93d42cdbb6c0a12a812b56a"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-w9tw"
I20240503 01:58:57.196915 3845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:57.224299 3845 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:57.225019 3845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:57.225247 3845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:57.225889 3845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:57.227079 3845 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:58:57.227222 3845 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:57.227344 3845 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:58:57.227388 3845 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:57.239321 3845 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:33129
I20240503 01:58:57.239336 3972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:33129 every 8 connection(s)
I20240503 01:58:57.240517 3845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 01:58:57.243541 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 3845
I20240503 01:58:57.243968 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:45035
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:35781
--webserver_port=42189
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 01:58:57.253741 3973 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35781
I20240503 01:58:57.254052 3973 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:57.254804 3973 heartbeater.cc:511] Master 127.0.99.190:35781 requested a full tablet report, sending...
I20240503 01:58:57.256078 2253 ts_manager.cc:194] Re-registered known tserver with Master: f0ced467f93d42cdbb6c0a12a812b56a (127.0.99.134:33129)
I20240503 01:58:57.256899 2253 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:57899
W20240503 01:58:57.376457 3976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:58:57.376811 3976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:58:57.376865 3976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:58:57.381225 3976 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:58:57.381332 3976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:58:57.381423 3976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 01:58:57.385670 3976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:45035
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=42189
--tserver_master_addrs=127.0.99.190:35781
--never_fsync=true
--heap_profile_path=/tmp/kudu.3976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:58:57.386735 3976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:58:57.387976 3976 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:58:57.399063 3984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:57.399883 3982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:58:57.400977 3981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:58:57.402259 3976 server_base.cc:1034] running on GCE node
I20240503 01:58:57.402923 3976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:58:57.403872 3976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:58:57.405081 3976 hybrid_clock.cc:648] HybridClock initialized: now 1714701537405052 us; error 47 us; skew 500 ppm
I20240503 01:58:57.411245 3976 webserver.cc:460] Webserver started at http://127.0.99.135:42189/ using document root <none> and password file <none>
I20240503 01:58:57.411906 3976 fs_manager.cc:359] Metadata directory not provided
I20240503 01:58:57.411996 3976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:58:57.416077 3976 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 01:58:57.418756 3990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:58:57.420349 3976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:58:57.420542 3976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "5372f615183544f8bffc0fe13f4a152f"
format_stamp: "Formatted at 2024-05-03 01:58:50 on dist-test-slave-w9tw"
I20240503 01:58:57.421125 3976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:58:57.438390 3976 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:58:57.439108 3976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:58:57.439255 3976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:58:57.439919 3976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:58:57.441316 3997 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240503 01:58:57.445816 3976 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 01:58:57.446000 3976 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20240503 01:58:57.446116 3976 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 01:58:57.449026 3976 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 01:58:57.449112 3976 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20240503 01:58:57.449455 3997 tablet_bootstrap.cc:492] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f: Bootstrap starting.
I20240503 01:58:57.468225 3976 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:45035
I20240503 01:58:57.468286 4104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:45035 every 8 connection(s)
I20240503 01:58:57.469527 3976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 01:58:57.472167 3997 log.cc:824] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:57.472286 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 3976
I20240503 01:58:57.477316 3997 tablet_bootstrap.cc:492] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:58:57.477741 3997 tablet_bootstrap.cc:492] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f: Bootstrap complete.
I20240503 01:58:57.479235 3997 ts_tablet_manager.cc:1367] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f: Time spent bootstrapping tablet: real 0.030s user 0.015s sys 0.012s
I20240503 01:58:57.483664 4105 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35781
I20240503 01:58:57.484020 4105 heartbeater.cc:464] Registering TS with master...
I20240503 01:58:57.483896 3997 raft_consensus.cc:348] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5372f615183544f8bffc0fe13f4a152f" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45035 } }
I20240503 01:58:57.484419 3997 raft_consensus.cc:729] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5372f615183544f8bffc0fe13f4a152f, State: Initialized, Role: FOLLOWER
I20240503 01:58:57.484994 4105 heartbeater.cc:511] Master 127.0.99.190:35781 requested a full tablet report, sending...
I20240503 01:58:57.484961 3997 consensus_queue.cc:260] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5372f615183544f8bffc0fe13f4a152f" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45035 } }
I20240503 01:58:57.485203 3997 raft_consensus.cc:388] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:58:57.485278 3997 raft_consensus.cc:482] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:58:57.485626 3997 raft_consensus.cc:3037] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:58:57.489071 3997 raft_consensus.cc:504] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5372f615183544f8bffc0fe13f4a152f" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45035 } }
I20240503 01:58:57.489756 3997 leader_election.cc:302] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5372f615183544f8bffc0fe13f4a152f; no voters:
I20240503 01:58:57.490286 3997 leader_election.cc:288] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f [CANDIDATE]: Term 2 election: Requested vote from peers
I20240503 01:58:57.490429 4110 raft_consensus.cc:2781] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:58:57.490679 4110 raft_consensus.cc:686] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f [term 2 LEADER]: Becoming Leader. State: Replica: 5372f615183544f8bffc0fe13f4a152f, State: Running, Role: LEADER
I20240503 01:58:57.491151 4110 consensus_queue.cc:237] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5372f615183544f8bffc0fe13f4a152f" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45035 } }
I20240503 01:58:57.491677 3997 ts_tablet_manager.cc:1398] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f: Time spent starting tablet: real 0.012s user 0.006s sys 0.006s
I20240503 01:58:57.492784 2253 ts_manager.cc:194] Re-registered known tserver with Master: 5372f615183544f8bffc0fe13f4a152f (127.0.99.135:45035)
I20240503 01:58:57.493893 2253 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:38633
I20240503 01:58:57.496778 4105 heartbeater.cc:503] Master 127.0.99.190:35781 was elected leader, sending a full tablet report...
I20240503 01:58:57.498222 2253 catalog_manager.cc:5484] T 3fc7ae29d38b4c539498cb946f1c9eaf P 5372f615183544f8bffc0fe13f4a152f reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "5372f615183544f8bffc0fe13f4a152f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5372f615183544f8bffc0fe13f4a152f" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45035 } } }
I20240503 01:58:57.591318 3580 heartbeater.cc:503] Master 127.0.99.190:35781 was elected leader, sending a full tablet report...
I20240503 01:58:57.710813 2379 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:58:57.713508 3776 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:57.714395 2510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:58:57.714637 3643 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:57.721876 4039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:58:57.723753 3907 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:57.728018 3514 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:57.820592 3711 heartbeater.cc:503] Master 127.0.99.190:35781 was elected leader, sending a full tablet report...
I20240503 01:58:57.871482 3776 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:57.873829 2379 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:58:57.875442 2510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:58:57.879709 3514 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:57.879753 3907 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:57.881690 3643 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:58:57.887135 4039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:58:57.953074 2530 consensus_queue.cc:237] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "65d5a1053fb5445486433fe66d82bdbe" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43855 } }
W20240503 01:58:57.960299 2465 consensus_peers.cc:485] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2 -> Peer 65d5a1053fb5445486433fe66d82bdbe (127.0.99.132:43855): Couldn't send request to peer 65d5a1053fb5445486433fe66d82bdbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 986d142f7be641ed9a9c338c41ba430a. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:58:58.021654 3842 heartbeater.cc:503] Master 127.0.99.190:35781 was elected leader, sending a full tablet report...
I20240503 01:58:58.259882 3973 heartbeater.cc:503] Master 127.0.99.190:35781 was elected leader, sending a full tablet report...
I20240503 01:58:58.561035 4219 ts_tablet_manager.cc:902] T 986d142f7be641ed9a9c338c41ba430a P 65d5a1053fb5445486433fe66d82bdbe: Initiating tablet copy from peer 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130:33335)
I20240503 01:58:58.561755 4219 tablet_copy_client.cc:308] T 986d142f7be641ed9a9c338c41ba430a P 65d5a1053fb5445486433fe66d82bdbe: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:33335
I20240503 01:58:58.566077 2550 tablet_copy_service.cc:140] P 86ed23734e434f8784885ebf9f1423b2: Received BeginTabletCopySession request for tablet 986d142f7be641ed9a9c338c41ba430a from peer 65d5a1053fb5445486433fe66d82bdbe ({username='slave'} at 127.0.99.132:47255)
I20240503 01:58:58.566231 2550 tablet_copy_service.cc:161] P 86ed23734e434f8784885ebf9f1423b2: Beginning new tablet copy session on tablet 986d142f7be641ed9a9c338c41ba430a from peer 65d5a1053fb5445486433fe66d82bdbe at {username='slave'} at 127.0.99.132:47255: session id = 65d5a1053fb5445486433fe66d82bdbe-986d142f7be641ed9a9c338c41ba430a
I20240503 01:58:58.567473 2550 tablet_copy_source_session.cc:205] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:58:58.571853 4219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 986d142f7be641ed9a9c338c41ba430a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:58.579797 4219 tablet_copy_client.cc:783] T 986d142f7be641ed9a9c338c41ba430a P 65d5a1053fb5445486433fe66d82bdbe: tablet copy: Starting download of 0 data blocks...
I20240503 01:58:58.580091 4219 tablet_copy_client.cc:647] T 986d142f7be641ed9a9c338c41ba430a P 65d5a1053fb5445486433fe66d82bdbe: tablet copy: Starting download of 1 WAL segments...
I20240503 01:58:58.582495 4219 tablet_copy_client.cc:523] T 986d142f7be641ed9a9c338c41ba430a P 65d5a1053fb5445486433fe66d82bdbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:58:58.584532 4219 tablet_bootstrap.cc:492] T 986d142f7be641ed9a9c338c41ba430a P 65d5a1053fb5445486433fe66d82bdbe: Bootstrap starting.
I20240503 01:58:58.590571 4219 log.cc:824] T 986d142f7be641ed9a9c338c41ba430a P 65d5a1053fb5445486433fe66d82bdbe: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:58.611991 4219 tablet_bootstrap.cc:492] T 986d142f7be641ed9a9c338c41ba430a P 65d5a1053fb5445486433fe66d82bdbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 01:58:58.612486 4219 tablet_bootstrap.cc:492] T 986d142f7be641ed9a9c338c41ba430a P 65d5a1053fb5445486433fe66d82bdbe: Bootstrap complete.
I20240503 01:58:58.612859 4219 ts_tablet_manager.cc:1367] T 986d142f7be641ed9a9c338c41ba430a P 65d5a1053fb5445486433fe66d82bdbe: Time spent bootstrapping tablet: real 0.028s user 0.020s sys 0.008s
I20240503 01:58:58.615281 4219 raft_consensus.cc:348] T 986d142f7be641ed9a9c338c41ba430a P 65d5a1053fb5445486433fe66d82bdbe [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:58:58.615942 4219 raft_consensus.cc:729] T 986d142f7be641ed9a9c338c41ba430a P 65d5a1053fb5445486433fe66d82bdbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65d5a1053fb5445486433fe66d82bdbe, State: Initialized, Role: FOLLOWER
I20240503 01:58:58.616267 4219 consensus_queue.cc:260] T 986d142f7be641ed9a9c338c41ba430a P 65d5a1053fb5445486433fe66d82bdbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "65d5a1053fb5445486433fe66d82bdbe" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43855 } }
I20240503 01:58:58.617273 4219 ts_tablet_manager.cc:1398] T 986d142f7be641ed9a9c338c41ba430a P 65d5a1053fb5445486433fe66d82bdbe: Time spent starting tablet: real 0.004s user 0.005s sys 0.001s
I20240503 01:58:58.618341 2550 tablet_copy_service.cc:342] P 86ed23734e434f8784885ebf9f1423b2: Request end of tablet copy session 65d5a1053fb5445486433fe66d82bdbe-986d142f7be641ed9a9c338c41ba430a received from {username='slave'} at 127.0.99.132:47255
I20240503 01:58:58.618494 2550 tablet_copy_service.cc:433] P 86ed23734e434f8784885ebf9f1423b2: ending tablet copy session 65d5a1053fb5445486433fe66d82bdbe-986d142f7be641ed9a9c338c41ba430a on tablet 986d142f7be641ed9a9c338c41ba430a with peer 65d5a1053fb5445486433fe66d82bdbe
I20240503 01:58:59.103332 3665 raft_consensus.cc:1203] T 986d142f7be641ed9a9c338c41ba430a P 65d5a1053fb5445486433fe66d82bdbe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:58:59.104791 4229 raft_consensus.cc:2932] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 65d5a1053fb5445486433fe66d82bdbe (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "65d5a1053fb5445486433fe66d82bdbe" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43855 } } }
I20240503 01:58:59.105420 3665 raft_consensus.cc:2932] T 986d142f7be641ed9a9c338c41ba430a P 65d5a1053fb5445486433fe66d82bdbe [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 65d5a1053fb5445486433fe66d82bdbe (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "65d5a1053fb5445486433fe66d82bdbe" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43855 } } }
I20240503 01:58:59.106671 4229 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:44874 (request call id 0) took 1154 ms. Trace:
I20240503 01:58:59.106815 4229 rpcz_store.cc:276] 0503 01:58:57.952134 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 01:58:57.952240 (+ 106us) service_pool.cc:225] Handling call
0503 01:58:57.954342 (+ 2102us) log.cc:842] Serialized 257 byte log entry
0503 01:58:59.106652 (+1152310us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":92,"raft.queue_time_us":177,"raft.run_cpu_time_us":616,"raft.run_wall_time_us":616,"spinlock_wait_cycles":525952,"thread_start_us":316,"threads_started":2,"wal-append.queue_time_us":375,"wal-append.run_cpu_time_us":585,"wal-append.run_wall_time_us":1000671}
I20240503 01:58:59.108705 2253 catalog_manager.cc:5484] T 986d142f7be641ed9a9c338c41ba430a P 65d5a1053fb5445486433fe66d82bdbe reported cstate change: config changed from index -1 to 4, VOTER 65d5a1053fb5445486433fe66d82bdbe (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "86ed23734e434f8784885ebf9f1423b2" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "65d5a1053fb5445486433fe66d82bdbe" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43855 } } }
I20240503 01:58:59.129665 2399 consensus_queue.cc:237] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "03ea675b68a74c1192fa1a96f89a494b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38459 } }
W20240503 01:58:59.136664 2334 consensus_peers.cc:485] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0 -> Peer 03ea675b68a74c1192fa1a96f89a494b (127.0.99.133:38459): Couldn't send request to peer 03ea675b68a74c1192fa1a96f89a494b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 358379b5f00a41d69db831501091e702. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:58:59.554718 4257 ts_tablet_manager.cc:902] T 358379b5f00a41d69db831501091e702 P 03ea675b68a74c1192fa1a96f89a494b: Initiating tablet copy from peer 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129:46305)
I20240503 01:58:59.555537 4257 tablet_copy_client.cc:308] T 358379b5f00a41d69db831501091e702 P 03ea675b68a74c1192fa1a96f89a494b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46305
I20240503 01:58:59.559423 2419 tablet_copy_service.cc:140] P 80a019d9d1c34ccd9487207f90e68ab0: Received BeginTabletCopySession request for tablet 358379b5f00a41d69db831501091e702 from peer 03ea675b68a74c1192fa1a96f89a494b ({username='slave'} at 127.0.99.133:57035)
I20240503 01:58:59.559557 2419 tablet_copy_service.cc:161] P 80a019d9d1c34ccd9487207f90e68ab0: Beginning new tablet copy session on tablet 358379b5f00a41d69db831501091e702 from peer 03ea675b68a74c1192fa1a96f89a494b at {username='slave'} at 127.0.99.133:57035: session id = 03ea675b68a74c1192fa1a96f89a494b-358379b5f00a41d69db831501091e702
I20240503 01:58:59.560729 2419 tablet_copy_source_session.cc:205] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:58:59.564323 4257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 358379b5f00a41d69db831501091e702. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:58:59.569975 4257 tablet_copy_client.cc:783] T 358379b5f00a41d69db831501091e702 P 03ea675b68a74c1192fa1a96f89a494b: tablet copy: Starting download of 0 data blocks...
I20240503 01:58:59.570183 4257 tablet_copy_client.cc:647] T 358379b5f00a41d69db831501091e702 P 03ea675b68a74c1192fa1a96f89a494b: tablet copy: Starting download of 1 WAL segments...
I20240503 01:58:59.572862 4257 tablet_copy_client.cc:523] T 358379b5f00a41d69db831501091e702 P 03ea675b68a74c1192fa1a96f89a494b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:58:59.574935 4257 tablet_bootstrap.cc:492] T 358379b5f00a41d69db831501091e702 P 03ea675b68a74c1192fa1a96f89a494b: Bootstrap starting.
I20240503 01:58:59.580756 4257 log.cc:824] T 358379b5f00a41d69db831501091e702 P 03ea675b68a74c1192fa1a96f89a494b: Log is configured to *not* fsync() on all Append() calls
I20240503 01:58:59.602334 4257 tablet_bootstrap.cc:492] T 358379b5f00a41d69db831501091e702 P 03ea675b68a74c1192fa1a96f89a494b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 01:58:59.602787 4257 tablet_bootstrap.cc:492] T 358379b5f00a41d69db831501091e702 P 03ea675b68a74c1192fa1a96f89a494b: Bootstrap complete.
I20240503 01:58:59.603114 4257 ts_tablet_manager.cc:1367] T 358379b5f00a41d69db831501091e702 P 03ea675b68a74c1192fa1a96f89a494b: Time spent bootstrapping tablet: real 0.028s user 0.024s sys 0.004s
I20240503 01:58:59.605253 4257 raft_consensus.cc:348] T 358379b5f00a41d69db831501091e702 P 03ea675b68a74c1192fa1a96f89a494b [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:58:59.605867 4257 raft_consensus.cc:729] T 358379b5f00a41d69db831501091e702 P 03ea675b68a74c1192fa1a96f89a494b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ea675b68a74c1192fa1a96f89a494b, State: Initialized, Role: FOLLOWER
I20240503 01:58:59.606190 4257 consensus_queue.cc:260] T 358379b5f00a41d69db831501091e702 P 03ea675b68a74c1192fa1a96f89a494b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "03ea675b68a74c1192fa1a96f89a494b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38459 } }
I20240503 01:58:59.607136 4257 ts_tablet_manager.cc:1398] T 358379b5f00a41d69db831501091e702 P 03ea675b68a74c1192fa1a96f89a494b: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 01:58:59.608466 2419 tablet_copy_service.cc:342] P 80a019d9d1c34ccd9487207f90e68ab0: Request end of tablet copy session 03ea675b68a74c1192fa1a96f89a494b-358379b5f00a41d69db831501091e702 received from {username='slave'} at 127.0.99.133:57035
I20240503 01:58:59.608597 2419 tablet_copy_service.cc:433] P 80a019d9d1c34ccd9487207f90e68ab0: ending tablet copy session 03ea675b68a74c1192fa1a96f89a494b-358379b5f00a41d69db831501091e702 on tablet 358379b5f00a41d69db831501091e702 with peer 03ea675b68a74c1192fa1a96f89a494b
I20240503 01:59:00.161186 3796 raft_consensus.cc:1203] T 358379b5f00a41d69db831501091e702 P 03ea675b68a74c1192fa1a96f89a494b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:00.162885 4265 raft_consensus.cc:2932] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 03ea675b68a74c1192fa1a96f89a494b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "03ea675b68a74c1192fa1a96f89a494b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38459 } } }
I20240503 01:59:00.163324 3796 raft_consensus.cc:2932] T 358379b5f00a41d69db831501091e702 P 03ea675b68a74c1192fa1a96f89a494b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 03ea675b68a74c1192fa1a96f89a494b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "03ea675b68a74c1192fa1a96f89a494b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38459 } } }
I20240503 01:59:00.164381 4265 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:47890 (request call id 0) took 1035 ms. Trace:
I20240503 01:59:00.164738 4265 rpcz_store.cc:276] 0503 01:58:59.128952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 01:58:59.129012 (+ 60us) service_pool.cc:225] Handling call
0503 01:58:59.130569 (+ 1557us) log.cc:842] Serialized 257 byte log entry
0503 01:59:00.164364 (+1033795us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":53,"raft.queue_time_us":207,"raft.run_cpu_time_us":342,"raft.run_wall_time_us":342,"thread_start_us":330,"threads_started":2,"wal-append.queue_time_us":335,"wal-append.run_cpu_time_us":324,"wal-append.run_wall_time_us":1000417}
I20240503 01:59:00.165654 2253 catalog_manager.cc:5484] T 358379b5f00a41d69db831501091e702 P 03ea675b68a74c1192fa1a96f89a494b reported cstate change: config changed from index -1 to 4, VOTER 03ea675b68a74c1192fa1a96f89a494b (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "80a019d9d1c34ccd9487207f90e68ab0" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "03ea675b68a74c1192fa1a96f89a494b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38459 } } }
I20240503 01:59:00.186007 2530 consensus_queue.cc:237] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "cc5fffa9b8844c6f9c948344d8081833" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45579 } }
W20240503 01:59:00.191546 2465 consensus_peers.cc:485] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2 -> Peer cc5fffa9b8844c6f9c948344d8081833 (127.0.99.131:45579): Couldn't send request to peer cc5fffa9b8844c6f9c948344d8081833. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a614380f92f64189ac5b5cdb42edf2e6. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:00.658013 4294 ts_tablet_manager.cc:902] T a614380f92f64189ac5b5cdb42edf2e6 P cc5fffa9b8844c6f9c948344d8081833: Initiating tablet copy from peer 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130:33335)
I20240503 01:59:00.658823 4294 tablet_copy_client.cc:308] T a614380f92f64189ac5b5cdb42edf2e6 P cc5fffa9b8844c6f9c948344d8081833: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:33335
I20240503 01:59:00.662735 2550 tablet_copy_service.cc:140] P 86ed23734e434f8784885ebf9f1423b2: Received BeginTabletCopySession request for tablet a614380f92f64189ac5b5cdb42edf2e6 from peer cc5fffa9b8844c6f9c948344d8081833 ({username='slave'} at 127.0.99.131:51599)
I20240503 01:59:00.663117 2550 tablet_copy_service.cc:161] P 86ed23734e434f8784885ebf9f1423b2: Beginning new tablet copy session on tablet a614380f92f64189ac5b5cdb42edf2e6 from peer cc5fffa9b8844c6f9c948344d8081833 at {username='slave'} at 127.0.99.131:51599: session id = cc5fffa9b8844c6f9c948344d8081833-a614380f92f64189ac5b5cdb42edf2e6
I20240503 01:59:00.664158 2550 tablet_copy_source_session.cc:205] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:00.667726 4294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a614380f92f64189ac5b5cdb42edf2e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:00.674252 4294 tablet_copy_client.cc:783] T a614380f92f64189ac5b5cdb42edf2e6 P cc5fffa9b8844c6f9c948344d8081833: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:00.674479 4294 tablet_copy_client.cc:647] T a614380f92f64189ac5b5cdb42edf2e6 P cc5fffa9b8844c6f9c948344d8081833: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:00.676307 4294 tablet_copy_client.cc:523] T a614380f92f64189ac5b5cdb42edf2e6 P cc5fffa9b8844c6f9c948344d8081833: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:00.678305 4294 tablet_bootstrap.cc:492] T a614380f92f64189ac5b5cdb42edf2e6 P cc5fffa9b8844c6f9c948344d8081833: Bootstrap starting.
I20240503 01:59:00.683938 4294 log.cc:824] T a614380f92f64189ac5b5cdb42edf2e6 P cc5fffa9b8844c6f9c948344d8081833: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:00.705837 4294 tablet_bootstrap.cc:492] T a614380f92f64189ac5b5cdb42edf2e6 P cc5fffa9b8844c6f9c948344d8081833: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 01:59:00.706302 4294 tablet_bootstrap.cc:492] T a614380f92f64189ac5b5cdb42edf2e6 P cc5fffa9b8844c6f9c948344d8081833: Bootstrap complete.
I20240503 01:59:00.706678 4294 ts_tablet_manager.cc:1367] T a614380f92f64189ac5b5cdb42edf2e6 P cc5fffa9b8844c6f9c948344d8081833: Time spent bootstrapping tablet: real 0.029s user 0.026s sys 0.004s
I20240503 01:59:00.709036 4294 raft_consensus.cc:348] T a614380f92f64189ac5b5cdb42edf2e6 P cc5fffa9b8844c6f9c948344d8081833 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:59:00.709893 4294 raft_consensus.cc:729] T a614380f92f64189ac5b5cdb42edf2e6 P cc5fffa9b8844c6f9c948344d8081833 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc5fffa9b8844c6f9c948344d8081833, State: Initialized, Role: FOLLOWER
I20240503 01:59:00.710261 4294 consensus_queue.cc:260] T a614380f92f64189ac5b5cdb42edf2e6 P cc5fffa9b8844c6f9c948344d8081833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "cc5fffa9b8844c6f9c948344d8081833" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45579 } }
I20240503 01:59:00.711269 4294 ts_tablet_manager.cc:1398] T a614380f92f64189ac5b5cdb42edf2e6 P cc5fffa9b8844c6f9c948344d8081833: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20240503 01:59:00.712378 2550 tablet_copy_service.cc:342] P 86ed23734e434f8784885ebf9f1423b2: Request end of tablet copy session cc5fffa9b8844c6f9c948344d8081833-a614380f92f64189ac5b5cdb42edf2e6 received from {username='slave'} at 127.0.99.131:51599
I20240503 01:59:00.712486 2550 tablet_copy_service.cc:433] P 86ed23734e434f8784885ebf9f1423b2: ending tablet copy session cc5fffa9b8844c6f9c948344d8081833-a614380f92f64189ac5b5cdb42edf2e6 on tablet a614380f92f64189ac5b5cdb42edf2e6 with peer cc5fffa9b8844c6f9c948344d8081833
I20240503 01:59:01.073894 3534 raft_consensus.cc:1203] T a614380f92f64189ac5b5cdb42edf2e6 P cc5fffa9b8844c6f9c948344d8081833 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:01.075454 4303 raft_consensus.cc:2932] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cc5fffa9b8844c6f9c948344d8081833 (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "cc5fffa9b8844c6f9c948344d8081833" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45579 } } }
I20240503 01:59:01.076333 3534 raft_consensus.cc:2932] T a614380f92f64189ac5b5cdb42edf2e6 P cc5fffa9b8844c6f9c948344d8081833 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cc5fffa9b8844c6f9c948344d8081833 (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "cc5fffa9b8844c6f9c948344d8081833" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45579 } } }
I20240503 01:59:01.079430 2255 catalog_manager.cc:5484] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2 reported cstate change: config changed from index -1 to 4, VOTER cc5fffa9b8844c6f9c948344d8081833 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "86ed23734e434f8784885ebf9f1423b2" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "cc5fffa9b8844c6f9c948344d8081833" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45579 } } }
I20240503 01:59:01.106657 2399 consensus_queue.cc:237] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "f0ced467f93d42cdbb6c0a12a812b56a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33129 } }
W20240503 01:59:01.113068 2332 consensus_peers.cc:485] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0 -> Peer f0ced467f93d42cdbb6c0a12a812b56a (127.0.99.134:33129): Couldn't send request to peer f0ced467f93d42cdbb6c0a12a812b56a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32f685df61f144628bbb19c89e36ee6e. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:01.673065 4328 ts_tablet_manager.cc:902] T 32f685df61f144628bbb19c89e36ee6e P f0ced467f93d42cdbb6c0a12a812b56a: Initiating tablet copy from peer 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129:46305)
I20240503 01:59:01.674099 4328 tablet_copy_client.cc:308] T 32f685df61f144628bbb19c89e36ee6e P f0ced467f93d42cdbb6c0a12a812b56a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46305
I20240503 01:59:01.678669 2419 tablet_copy_service.cc:140] P 80a019d9d1c34ccd9487207f90e68ab0: Received BeginTabletCopySession request for tablet 32f685df61f144628bbb19c89e36ee6e from peer f0ced467f93d42cdbb6c0a12a812b56a ({username='slave'} at 127.0.99.134:53317)
I20240503 01:59:01.678807 2419 tablet_copy_service.cc:161] P 80a019d9d1c34ccd9487207f90e68ab0: Beginning new tablet copy session on tablet 32f685df61f144628bbb19c89e36ee6e from peer f0ced467f93d42cdbb6c0a12a812b56a at {username='slave'} at 127.0.99.134:53317: session id = f0ced467f93d42cdbb6c0a12a812b56a-32f685df61f144628bbb19c89e36ee6e
I20240503 01:59:01.679615 2419 tablet_copy_source_session.cc:205] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:01.682818 4328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32f685df61f144628bbb19c89e36ee6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:01.688169 4328 tablet_copy_client.cc:783] T 32f685df61f144628bbb19c89e36ee6e P f0ced467f93d42cdbb6c0a12a812b56a: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:01.688385 4328 tablet_copy_client.cc:647] T 32f685df61f144628bbb19c89e36ee6e P f0ced467f93d42cdbb6c0a12a812b56a: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:01.690292 4328 tablet_copy_client.cc:523] T 32f685df61f144628bbb19c89e36ee6e P f0ced467f93d42cdbb6c0a12a812b56a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:01.692149 4328 tablet_bootstrap.cc:492] T 32f685df61f144628bbb19c89e36ee6e P f0ced467f93d42cdbb6c0a12a812b56a: Bootstrap starting.
I20240503 01:59:01.698388 4328 log.cc:824] T 32f685df61f144628bbb19c89e36ee6e P f0ced467f93d42cdbb6c0a12a812b56a: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:01.719130 4328 tablet_bootstrap.cc:492] T 32f685df61f144628bbb19c89e36ee6e P f0ced467f93d42cdbb6c0a12a812b56a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 01:59:01.719552 4328 tablet_bootstrap.cc:492] T 32f685df61f144628bbb19c89e36ee6e P f0ced467f93d42cdbb6c0a12a812b56a: Bootstrap complete.
I20240503 01:59:01.719951 4328 ts_tablet_manager.cc:1367] T 32f685df61f144628bbb19c89e36ee6e P f0ced467f93d42cdbb6c0a12a812b56a: Time spent bootstrapping tablet: real 0.028s user 0.027s sys 0.001s
I20240503 01:59:01.721994 4328 raft_consensus.cc:348] T 32f685df61f144628bbb19c89e36ee6e P f0ced467f93d42cdbb6c0a12a812b56a [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:59:01.722609 4328 raft_consensus.cc:729] T 32f685df61f144628bbb19c89e36ee6e P f0ced467f93d42cdbb6c0a12a812b56a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0ced467f93d42cdbb6c0a12a812b56a, State: Initialized, Role: FOLLOWER
I20240503 01:59:01.722925 4328 consensus_queue.cc:260] T 32f685df61f144628bbb19c89e36ee6e P f0ced467f93d42cdbb6c0a12a812b56a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "f0ced467f93d42cdbb6c0a12a812b56a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33129 } }
I20240503 01:59:01.723819 4328 ts_tablet_manager.cc:1398] T 32f685df61f144628bbb19c89e36ee6e P f0ced467f93d42cdbb6c0a12a812b56a: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240503 01:59:01.724655 2419 tablet_copy_service.cc:342] P 80a019d9d1c34ccd9487207f90e68ab0: Request end of tablet copy session f0ced467f93d42cdbb6c0a12a812b56a-32f685df61f144628bbb19c89e36ee6e received from {username='slave'} at 127.0.99.134:53317
I20240503 01:59:01.724753 2419 tablet_copy_service.cc:433] P 80a019d9d1c34ccd9487207f90e68ab0: ending tablet copy session f0ced467f93d42cdbb6c0a12a812b56a-32f685df61f144628bbb19c89e36ee6e on tablet 32f685df61f144628bbb19c89e36ee6e with peer f0ced467f93d42cdbb6c0a12a812b56a
I20240503 01:59:02.101243 3927 raft_consensus.cc:1203] T 32f685df61f144628bbb19c89e36ee6e P f0ced467f93d42cdbb6c0a12a812b56a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:02.102473 4337 raft_consensus.cc:2932] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0ced467f93d42cdbb6c0a12a812b56a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "f0ced467f93d42cdbb6c0a12a812b56a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33129 } } }
I20240503 01:59:02.103030 3927 raft_consensus.cc:2932] T 32f685df61f144628bbb19c89e36ee6e P f0ced467f93d42cdbb6c0a12a812b56a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0ced467f93d42cdbb6c0a12a812b56a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "f0ced467f93d42cdbb6c0a12a812b56a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33129 } } }
I20240503 01:59:02.105502 2255 catalog_manager.cc:5484] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0 reported cstate change: config changed from index -1 to 4, VOTER f0ced467f93d42cdbb6c0a12a812b56a (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "80a019d9d1c34ccd9487207f90e68ab0" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "f0ced467f93d42cdbb6c0a12a812b56a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33129 } } }
I20240503 01:59:02.124485 2530 consensus_queue.cc:237] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "5372f615183544f8bffc0fe13f4a152f" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45035 } }
W20240503 01:59:02.129786 2465 consensus_peers.cc:485] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2 -> Peer 5372f615183544f8bffc0fe13f4a152f (127.0.99.135:45035): Couldn't send request to peer 5372f615183544f8bffc0fe13f4a152f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 833d9ce859f84b499e667d86965d9fff. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:02.571909 4363 ts_tablet_manager.cc:902] T 833d9ce859f84b499e667d86965d9fff P 5372f615183544f8bffc0fe13f4a152f: Initiating tablet copy from peer 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130:33335)
I20240503 01:59:02.572710 4363 tablet_copy_client.cc:308] T 833d9ce859f84b499e667d86965d9fff P 5372f615183544f8bffc0fe13f4a152f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:33335
I20240503 01:59:02.577343 2550 tablet_copy_service.cc:140] P 86ed23734e434f8784885ebf9f1423b2: Received BeginTabletCopySession request for tablet 833d9ce859f84b499e667d86965d9fff from peer 5372f615183544f8bffc0fe13f4a152f ({username='slave'} at 127.0.99.135:39981)
I20240503 01:59:02.577507 2550 tablet_copy_service.cc:161] P 86ed23734e434f8784885ebf9f1423b2: Beginning new tablet copy session on tablet 833d9ce859f84b499e667d86965d9fff from peer 5372f615183544f8bffc0fe13f4a152f at {username='slave'} at 127.0.99.135:39981: session id = 5372f615183544f8bffc0fe13f4a152f-833d9ce859f84b499e667d86965d9fff
I20240503 01:59:02.578483 2550 tablet_copy_source_session.cc:205] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:02.580945 4363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 833d9ce859f84b499e667d86965d9fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:02.583963 4363 tablet_copy_client.cc:783] T 833d9ce859f84b499e667d86965d9fff P 5372f615183544f8bffc0fe13f4a152f: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:02.584131 4363 tablet_copy_client.cc:647] T 833d9ce859f84b499e667d86965d9fff P 5372f615183544f8bffc0fe13f4a152f: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:02.585881 4363 tablet_copy_client.cc:523] T 833d9ce859f84b499e667d86965d9fff P 5372f615183544f8bffc0fe13f4a152f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:02.638115 4363 tablet_bootstrap.cc:492] T 833d9ce859f84b499e667d86965d9fff P 5372f615183544f8bffc0fe13f4a152f: Bootstrap starting.
I20240503 01:59:02.659226 4363 tablet_bootstrap.cc:492] T 833d9ce859f84b499e667d86965d9fff P 5372f615183544f8bffc0fe13f4a152f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 01:59:02.659626 4363 tablet_bootstrap.cc:492] T 833d9ce859f84b499e667d86965d9fff P 5372f615183544f8bffc0fe13f4a152f: Bootstrap complete.
I20240503 01:59:02.659804 4363 ts_tablet_manager.cc:1367] T 833d9ce859f84b499e667d86965d9fff P 5372f615183544f8bffc0fe13f4a152f: Time spent bootstrapping tablet: real 0.022s user 0.022s sys 0.000s
I20240503 01:59:02.660305 4363 raft_consensus.cc:348] T 833d9ce859f84b499e667d86965d9fff P 5372f615183544f8bffc0fe13f4a152f [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:59:02.660619 4363 raft_consensus.cc:729] T 833d9ce859f84b499e667d86965d9fff P 5372f615183544f8bffc0fe13f4a152f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5372f615183544f8bffc0fe13f4a152f, State: Initialized, Role: FOLLOWER
I20240503 01:59:02.660737 4363 consensus_queue.cc:260] T 833d9ce859f84b499e667d86965d9fff P 5372f615183544f8bffc0fe13f4a152f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "5372f615183544f8bffc0fe13f4a152f" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45035 } }
I20240503 01:59:02.661073 4363 ts_tablet_manager.cc:1398] T 833d9ce859f84b499e667d86965d9fff P 5372f615183544f8bffc0fe13f4a152f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:02.661761 2550 tablet_copy_service.cc:342] P 86ed23734e434f8784885ebf9f1423b2: Request end of tablet copy session 5372f615183544f8bffc0fe13f4a152f-833d9ce859f84b499e667d86965d9fff received from {username='slave'} at 127.0.99.135:39981
I20240503 01:59:02.661855 2550 tablet_copy_service.cc:433] P 86ed23734e434f8784885ebf9f1423b2: ending tablet copy session 5372f615183544f8bffc0fe13f4a152f-833d9ce859f84b499e667d86965d9fff on tablet 833d9ce859f84b499e667d86965d9fff with peer 5372f615183544f8bffc0fe13f4a152f
I20240503 01:59:03.192678 4059 raft_consensus.cc:1203] T 833d9ce859f84b499e667d86965d9fff P 5372f615183544f8bffc0fe13f4a152f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:03.194393 4372 raft_consensus.cc:2932] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5372f615183544f8bffc0fe13f4a152f (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "5372f615183544f8bffc0fe13f4a152f" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45035 } } }
I20240503 01:59:03.195013 4059 raft_consensus.cc:2932] T 833d9ce859f84b499e667d86965d9fff P 5372f615183544f8bffc0fe13f4a152f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5372f615183544f8bffc0fe13f4a152f (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "5372f615183544f8bffc0fe13f4a152f" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45035 } } }
I20240503 01:59:03.196415 4372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:52930 (request call id 0) took 1072 ms. Trace:
I20240503 01:59:03.196575 4372 rpcz_store.cc:276] 0503 01:59:02.124119 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 01:59:02.124185 (+ 66us) service_pool.cc:225] Handling call
0503 01:59:02.124989 (+ 804us) log.cc:842] Serialized 257 byte log entry
0503 01:59:03.196390 (+1071401us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":55,"raft.queue_time_us":43,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":108,"thread_start_us":65,"threads_started":1,"wal-append.queue_time_us":138,"wal-append.run_cpu_time_us":357,"wal-append.run_wall_time_us":1000450}
I20240503 01:59:03.198210 2253 catalog_manager.cc:5484] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2 reported cstate change: config changed from index -1 to 4, VOTER 5372f615183544f8bffc0fe13f4a152f (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "86ed23734e434f8784885ebf9f1423b2" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "5372f615183544f8bffc0fe13f4a152f" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45035 } } }
I20240503 01:59:03.221434 2399 consensus_queue.cc:237] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "65d5a1053fb5445486433fe66d82bdbe" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43855 } }
W20240503 01:59:03.226743 2334 consensus_peers.cc:485] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0 -> Peer 65d5a1053fb5445486433fe66d82bdbe (127.0.99.132:43855): Couldn't send request to peer 65d5a1053fb5445486433fe66d82bdbe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae344bce9aeb452b848accc96ddbb1ac. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:03.815343 4399 ts_tablet_manager.cc:902] T ae344bce9aeb452b848accc96ddbb1ac P 65d5a1053fb5445486433fe66d82bdbe: Initiating tablet copy from peer 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129:46305)
I20240503 01:59:03.815973 4399 tablet_copy_client.cc:308] T ae344bce9aeb452b848accc96ddbb1ac P 65d5a1053fb5445486433fe66d82bdbe: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46305
I20240503 01:59:03.820071 2419 tablet_copy_service.cc:140] P 80a019d9d1c34ccd9487207f90e68ab0: Received BeginTabletCopySession request for tablet ae344bce9aeb452b848accc96ddbb1ac from peer 65d5a1053fb5445486433fe66d82bdbe ({username='slave'} at 127.0.99.132:37903)
I20240503 01:59:03.820216 2419 tablet_copy_service.cc:161] P 80a019d9d1c34ccd9487207f90e68ab0: Beginning new tablet copy session on tablet ae344bce9aeb452b848accc96ddbb1ac from peer 65d5a1053fb5445486433fe66d82bdbe at {username='slave'} at 127.0.99.132:37903: session id = 65d5a1053fb5445486433fe66d82bdbe-ae344bce9aeb452b848accc96ddbb1ac
I20240503 01:59:03.821043 2419 tablet_copy_source_session.cc:205] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:03.822037 4399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae344bce9aeb452b848accc96ddbb1ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:03.824259 4399 tablet_copy_client.cc:783] T ae344bce9aeb452b848accc96ddbb1ac P 65d5a1053fb5445486433fe66d82bdbe: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:03.824420 4399 tablet_copy_client.cc:647] T ae344bce9aeb452b848accc96ddbb1ac P 65d5a1053fb5445486433fe66d82bdbe: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:03.825875 4399 tablet_copy_client.cc:523] T ae344bce9aeb452b848accc96ddbb1ac P 65d5a1053fb5445486433fe66d82bdbe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:03.827168 4399 tablet_bootstrap.cc:492] T ae344bce9aeb452b848accc96ddbb1ac P 65d5a1053fb5445486433fe66d82bdbe: Bootstrap starting.
I20240503 01:59:03.848531 4399 tablet_bootstrap.cc:492] T ae344bce9aeb452b848accc96ddbb1ac P 65d5a1053fb5445486433fe66d82bdbe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 01:59:03.849112 4399 tablet_bootstrap.cc:492] T ae344bce9aeb452b848accc96ddbb1ac P 65d5a1053fb5445486433fe66d82bdbe: Bootstrap complete.
I20240503 01:59:03.849391 4399 ts_tablet_manager.cc:1367] T ae344bce9aeb452b848accc96ddbb1ac P 65d5a1053fb5445486433fe66d82bdbe: Time spent bootstrapping tablet: real 0.022s user 0.020s sys 0.000s
I20240503 01:59:03.850301 4399 raft_consensus.cc:348] T ae344bce9aeb452b848accc96ddbb1ac P 65d5a1053fb5445486433fe66d82bdbe [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:59:03.850618 4399 raft_consensus.cc:729] T ae344bce9aeb452b848accc96ddbb1ac P 65d5a1053fb5445486433fe66d82bdbe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65d5a1053fb5445486433fe66d82bdbe, State: Initialized, Role: FOLLOWER
I20240503 01:59:03.850754 4399 consensus_queue.cc:260] T ae344bce9aeb452b848accc96ddbb1ac P 65d5a1053fb5445486433fe66d82bdbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "65d5a1053fb5445486433fe66d82bdbe" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43855 } }
I20240503 01:59:03.851181 4399 ts_tablet_manager.cc:1398] T ae344bce9aeb452b848accc96ddbb1ac P 65d5a1053fb5445486433fe66d82bdbe: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 01:59:03.851938 2419 tablet_copy_service.cc:342] P 80a019d9d1c34ccd9487207f90e68ab0: Request end of tablet copy session 65d5a1053fb5445486433fe66d82bdbe-ae344bce9aeb452b848accc96ddbb1ac received from {username='slave'} at 127.0.99.132:37903
I20240503 01:59:03.852075 2419 tablet_copy_service.cc:433] P 80a019d9d1c34ccd9487207f90e68ab0: ending tablet copy session 65d5a1053fb5445486433fe66d82bdbe-ae344bce9aeb452b848accc96ddbb1ac on tablet ae344bce9aeb452b848accc96ddbb1ac with peer 65d5a1053fb5445486433fe66d82bdbe
I20240503 01:59:04.364955 3665 raft_consensus.cc:1203] T ae344bce9aeb452b848accc96ddbb1ac P 65d5a1053fb5445486433fe66d82bdbe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:04.365926 4409 raft_consensus.cc:2932] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 65d5a1053fb5445486433fe66d82bdbe (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "65d5a1053fb5445486433fe66d82bdbe" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43855 } } }
I20240503 01:59:04.366482 3665 raft_consensus.cc:2932] T ae344bce9aeb452b848accc96ddbb1ac P 65d5a1053fb5445486433fe66d82bdbe [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 65d5a1053fb5445486433fe66d82bdbe (127.0.99.132) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "65d5a1053fb5445486433fe66d82bdbe" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43855 } } }
I20240503 01:59:04.367606 4409 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:43280 (request call id 0) took 1146 ms. Trace:
I20240503 01:59:04.367725 4409 rpcz_store.cc:276] 0503 01:59:03.220934 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 01:59:03.221038 (+ 104us) service_pool.cc:225] Handling call
0503 01:59:03.222004 (+ 966us) log.cc:842] Serialized 257 byte log entry
0503 01:59:04.367588 (+1145584us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":68,"raft.queue_time_us":94,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":121,"thread_start_us":86,"threads_started":1,"wal-append.queue_time_us":223,"wal-append.run_cpu_time_us":509,"wal-append.run_wall_time_us":1000587}
I20240503 01:59:04.369071 2253 catalog_manager.cc:5484] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0 reported cstate change: config changed from index -1 to 4, VOTER 65d5a1053fb5445486433fe66d82bdbe (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "80a019d9d1c34ccd9487207f90e68ab0" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "65d5a1053fb5445486433fe66d82bdbe" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43855 } } }
I20240503 01:59:04.392894 2399 consensus_queue.cc:237] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "03ea675b68a74c1192fa1a96f89a494b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38459 } }
W20240503 01:59:04.394502 2334 consensus_peers.cc:485] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0 -> Peer 03ea675b68a74c1192fa1a96f89a494b (127.0.99.133:38459): Couldn't send request to peer 03ea675b68a74c1192fa1a96f89a494b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49706355c8864e638e187eee7711cac3. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:04.936107 4434 ts_tablet_manager.cc:902] T 49706355c8864e638e187eee7711cac3 P 03ea675b68a74c1192fa1a96f89a494b: Initiating tablet copy from peer 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129:46305)
I20240503 01:59:04.936892 4434 tablet_copy_client.cc:308] T 49706355c8864e638e187eee7711cac3 P 03ea675b68a74c1192fa1a96f89a494b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46305
I20240503 01:59:04.937768 2419 tablet_copy_service.cc:140] P 80a019d9d1c34ccd9487207f90e68ab0: Received BeginTabletCopySession request for tablet 49706355c8864e638e187eee7711cac3 from peer 03ea675b68a74c1192fa1a96f89a494b ({username='slave'} at 127.0.99.133:57035)
I20240503 01:59:04.937887 2419 tablet_copy_service.cc:161] P 80a019d9d1c34ccd9487207f90e68ab0: Beginning new tablet copy session on tablet 49706355c8864e638e187eee7711cac3 from peer 03ea675b68a74c1192fa1a96f89a494b at {username='slave'} at 127.0.99.133:57035: session id = 03ea675b68a74c1192fa1a96f89a494b-49706355c8864e638e187eee7711cac3
I20240503 01:59:04.938721 2419 tablet_copy_source_session.cc:205] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:04.939720 4434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49706355c8864e638e187eee7711cac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:04.941551 4434 tablet_copy_client.cc:783] T 49706355c8864e638e187eee7711cac3 P 03ea675b68a74c1192fa1a96f89a494b: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:04.941728 4434 tablet_copy_client.cc:647] T 49706355c8864e638e187eee7711cac3 P 03ea675b68a74c1192fa1a96f89a494b: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:04.943378 4434 tablet_copy_client.cc:523] T 49706355c8864e638e187eee7711cac3 P 03ea675b68a74c1192fa1a96f89a494b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:04.944885 4434 tablet_bootstrap.cc:492] T 49706355c8864e638e187eee7711cac3 P 03ea675b68a74c1192fa1a96f89a494b: Bootstrap starting.
I20240503 01:59:04.965245 4434 tablet_bootstrap.cc:492] T 49706355c8864e638e187eee7711cac3 P 03ea675b68a74c1192fa1a96f89a494b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 01:59:04.965602 4434 tablet_bootstrap.cc:492] T 49706355c8864e638e187eee7711cac3 P 03ea675b68a74c1192fa1a96f89a494b: Bootstrap complete.
I20240503 01:59:04.965780 4434 ts_tablet_manager.cc:1367] T 49706355c8864e638e187eee7711cac3 P 03ea675b68a74c1192fa1a96f89a494b: Time spent bootstrapping tablet: real 0.021s user 0.019s sys 0.000s
I20240503 01:59:04.966380 4434 raft_consensus.cc:348] T 49706355c8864e638e187eee7711cac3 P 03ea675b68a74c1192fa1a96f89a494b [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:59:04.966643 4434 raft_consensus.cc:729] T 49706355c8864e638e187eee7711cac3 P 03ea675b68a74c1192fa1a96f89a494b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03ea675b68a74c1192fa1a96f89a494b, State: Initialized, Role: FOLLOWER
I20240503 01:59:04.966769 4434 consensus_queue.cc:260] T 49706355c8864e638e187eee7711cac3 P 03ea675b68a74c1192fa1a96f89a494b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "03ea675b68a74c1192fa1a96f89a494b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38459 } }
I20240503 01:59:04.967197 4434 ts_tablet_manager.cc:1398] T 49706355c8864e638e187eee7711cac3 P 03ea675b68a74c1192fa1a96f89a494b: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 01:59:04.967890 2419 tablet_copy_service.cc:342] P 80a019d9d1c34ccd9487207f90e68ab0: Request end of tablet copy session 03ea675b68a74c1192fa1a96f89a494b-49706355c8864e638e187eee7711cac3 received from {username='slave'} at 127.0.99.133:57035
I20240503 01:59:04.967972 2419 tablet_copy_service.cc:433] P 80a019d9d1c34ccd9487207f90e68ab0: ending tablet copy session 03ea675b68a74c1192fa1a96f89a494b-49706355c8864e638e187eee7711cac3 on tablet 49706355c8864e638e187eee7711cac3 with peer 03ea675b68a74c1192fa1a96f89a494b
I20240503 01:59:05.551967 3796 raft_consensus.cc:1203] T 49706355c8864e638e187eee7711cac3 P 03ea675b68a74c1192fa1a96f89a494b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:05.553264 4442 raft_consensus.cc:2932] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 03ea675b68a74c1192fa1a96f89a494b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "03ea675b68a74c1192fa1a96f89a494b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38459 } } }
I20240503 01:59:05.553718 3796 raft_consensus.cc:2932] T 49706355c8864e638e187eee7711cac3 P 03ea675b68a74c1192fa1a96f89a494b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 03ea675b68a74c1192fa1a96f89a494b (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "03ea675b68a74c1192fa1a96f89a494b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38459 } } }
I20240503 01:59:05.554818 4442 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:43308 (request call id 0) took 1162 ms. Trace:
I20240503 01:59:05.554952 4442 rpcz_store.cc:276] 0503 01:59:04.392485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 01:59:04.392565 (+ 80us) service_pool.cc:225] Handling call
0503 01:59:04.393300 (+ 735us) log.cc:842] Serialized 257 byte log entry
0503 01:59:05.554800 (+1161500us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":50,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":159,"spinlock_wait_cycles":12160,"thread_start_us":85,"threads_started":1,"wal-append.queue_time_us":202,"wal-append.run_cpu_time_us":368,"wal-append.run_wall_time_us":1000469}
I20240503 01:59:05.556581 2252 catalog_manager.cc:5484] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0 reported cstate change: config changed from index -1 to 4, VOTER 03ea675b68a74c1192fa1a96f89a494b (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "80a019d9d1c34ccd9487207f90e68ab0" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "03ea675b68a74c1192fa1a96f89a494b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38459 } } }
I20240503 01:59:05.579110 2530 consensus_queue.cc:237] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "cc5fffa9b8844c6f9c948344d8081833" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45579 } }
W20240503 01:59:05.580741 2465 consensus_peers.cc:485] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2 -> Peer cc5fffa9b8844c6f9c948344d8081833 (127.0.99.131:45579): Couldn't send request to peer cc5fffa9b8844c6f9c948344d8081833. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5c289e9ad385459aa89fa32637a42f36. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:05.997279 4467 ts_tablet_manager.cc:902] T 5c289e9ad385459aa89fa32637a42f36 P cc5fffa9b8844c6f9c948344d8081833: Initiating tablet copy from peer 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130:33335)
I20240503 01:59:05.997817 4467 tablet_copy_client.cc:308] T 5c289e9ad385459aa89fa32637a42f36 P cc5fffa9b8844c6f9c948344d8081833: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:33335
I20240503 01:59:05.998557 2550 tablet_copy_service.cc:140] P 86ed23734e434f8784885ebf9f1423b2: Received BeginTabletCopySession request for tablet 5c289e9ad385459aa89fa32637a42f36 from peer cc5fffa9b8844c6f9c948344d8081833 ({username='slave'} at 127.0.99.131:51599)
I20240503 01:59:05.998729 2550 tablet_copy_service.cc:161] P 86ed23734e434f8784885ebf9f1423b2: Beginning new tablet copy session on tablet 5c289e9ad385459aa89fa32637a42f36 from peer cc5fffa9b8844c6f9c948344d8081833 at {username='slave'} at 127.0.99.131:51599: session id = cc5fffa9b8844c6f9c948344d8081833-5c289e9ad385459aa89fa32637a42f36
I20240503 01:59:05.999611 2550 tablet_copy_source_session.cc:205] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:06.000735 4467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c289e9ad385459aa89fa32637a42f36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:06.002807 4467 tablet_copy_client.cc:783] T 5c289e9ad385459aa89fa32637a42f36 P cc5fffa9b8844c6f9c948344d8081833: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:06.002997 4467 tablet_copy_client.cc:647] T 5c289e9ad385459aa89fa32637a42f36 P cc5fffa9b8844c6f9c948344d8081833: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:06.004823 4467 tablet_copy_client.cc:523] T 5c289e9ad385459aa89fa32637a42f36 P cc5fffa9b8844c6f9c948344d8081833: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:06.006124 4467 tablet_bootstrap.cc:492] T 5c289e9ad385459aa89fa32637a42f36 P cc5fffa9b8844c6f9c948344d8081833: Bootstrap starting.
I20240503 01:59:06.026564 4467 tablet_bootstrap.cc:492] T 5c289e9ad385459aa89fa32637a42f36 P cc5fffa9b8844c6f9c948344d8081833: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 01:59:06.026911 4467 tablet_bootstrap.cc:492] T 5c289e9ad385459aa89fa32637a42f36 P cc5fffa9b8844c6f9c948344d8081833: Bootstrap complete.
I20240503 01:59:06.027110 4467 ts_tablet_manager.cc:1367] T 5c289e9ad385459aa89fa32637a42f36 P cc5fffa9b8844c6f9c948344d8081833: Time spent bootstrapping tablet: real 0.021s user 0.021s sys 0.000s
I20240503 01:59:06.027777 4467 raft_consensus.cc:348] T 5c289e9ad385459aa89fa32637a42f36 P cc5fffa9b8844c6f9c948344d8081833 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:59:06.028003 4467 raft_consensus.cc:729] T 5c289e9ad385459aa89fa32637a42f36 P cc5fffa9b8844c6f9c948344d8081833 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc5fffa9b8844c6f9c948344d8081833, State: Initialized, Role: FOLLOWER
I20240503 01:59:06.028111 4467 consensus_queue.cc:260] T 5c289e9ad385459aa89fa32637a42f36 P cc5fffa9b8844c6f9c948344d8081833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "cc5fffa9b8844c6f9c948344d8081833" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45579 } }
I20240503 01:59:06.028486 4467 ts_tablet_manager.cc:1398] T 5c289e9ad385459aa89fa32637a42f36 P cc5fffa9b8844c6f9c948344d8081833: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:06.029284 2550 tablet_copy_service.cc:342] P 86ed23734e434f8784885ebf9f1423b2: Request end of tablet copy session cc5fffa9b8844c6f9c948344d8081833-5c289e9ad385459aa89fa32637a42f36 received from {username='slave'} at 127.0.99.131:51599
I20240503 01:59:06.029433 2550 tablet_copy_service.cc:433] P 86ed23734e434f8784885ebf9f1423b2: ending tablet copy session cc5fffa9b8844c6f9c948344d8081833-5c289e9ad385459aa89fa32637a42f36 on tablet 5c289e9ad385459aa89fa32637a42f36 with peer cc5fffa9b8844c6f9c948344d8081833
I20240503 01:59:06.387643 3534 raft_consensus.cc:1203] T 5c289e9ad385459aa89fa32637a42f36 P cc5fffa9b8844c6f9c948344d8081833 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:06.388767 4474 raft_consensus.cc:2932] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cc5fffa9b8844c6f9c948344d8081833 (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "cc5fffa9b8844c6f9c948344d8081833" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45579 } } }
I20240503 01:59:06.389209 3534 raft_consensus.cc:2932] T 5c289e9ad385459aa89fa32637a42f36 P cc5fffa9b8844c6f9c948344d8081833 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cc5fffa9b8844c6f9c948344d8081833 (127.0.99.131) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "cc5fffa9b8844c6f9c948344d8081833" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45579 } } }
I20240503 01:59:06.392493 2252 catalog_manager.cc:5484] T 5c289e9ad385459aa89fa32637a42f36 P cc5fffa9b8844c6f9c948344d8081833 reported cstate change: config changed from index -1 to 4, VOTER cc5fffa9b8844c6f9c948344d8081833 (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "86ed23734e434f8784885ebf9f1423b2" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "cc5fffa9b8844c6f9c948344d8081833" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45579 } } }
I20240503 01:59:06.419716 2399 consensus_queue.cc:237] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "f0ced467f93d42cdbb6c0a12a812b56a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33129 } }
W20240503 01:59:06.421388 2332 consensus_peers.cc:485] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0 -> Peer f0ced467f93d42cdbb6c0a12a812b56a (127.0.99.134:33129): Couldn't send request to peer f0ced467f93d42cdbb6c0a12a812b56a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0bce60681a34a96a83284c762d6feee. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:06.929136 4497 ts_tablet_manager.cc:902] T c0bce60681a34a96a83284c762d6feee P f0ced467f93d42cdbb6c0a12a812b56a: Initiating tablet copy from peer 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129:46305)
I20240503 01:59:06.929697 4497 tablet_copy_client.cc:308] T c0bce60681a34a96a83284c762d6feee P f0ced467f93d42cdbb6c0a12a812b56a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46305
I20240503 01:59:06.930357 2419 tablet_copy_service.cc:140] P 80a019d9d1c34ccd9487207f90e68ab0: Received BeginTabletCopySession request for tablet c0bce60681a34a96a83284c762d6feee from peer f0ced467f93d42cdbb6c0a12a812b56a ({username='slave'} at 127.0.99.134:53317)
I20240503 01:59:06.930477 2419 tablet_copy_service.cc:161] P 80a019d9d1c34ccd9487207f90e68ab0: Beginning new tablet copy session on tablet c0bce60681a34a96a83284c762d6feee from peer f0ced467f93d42cdbb6c0a12a812b56a at {username='slave'} at 127.0.99.134:53317: session id = f0ced467f93d42cdbb6c0a12a812b56a-c0bce60681a34a96a83284c762d6feee
I20240503 01:59:06.931268 2419 tablet_copy_source_session.cc:205] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:06.932477 4497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0bce60681a34a96a83284c762d6feee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:06.935283 4497 tablet_copy_client.cc:783] T c0bce60681a34a96a83284c762d6feee P f0ced467f93d42cdbb6c0a12a812b56a: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:06.935561 4497 tablet_copy_client.cc:647] T c0bce60681a34a96a83284c762d6feee P f0ced467f93d42cdbb6c0a12a812b56a: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:06.937348 4497 tablet_copy_client.cc:523] T c0bce60681a34a96a83284c762d6feee P f0ced467f93d42cdbb6c0a12a812b56a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:06.938791 4497 tablet_bootstrap.cc:492] T c0bce60681a34a96a83284c762d6feee P f0ced467f93d42cdbb6c0a12a812b56a: Bootstrap starting.
I20240503 01:59:06.958889 4497 tablet_bootstrap.cc:492] T c0bce60681a34a96a83284c762d6feee P f0ced467f93d42cdbb6c0a12a812b56a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 01:59:06.959295 4497 tablet_bootstrap.cc:492] T c0bce60681a34a96a83284c762d6feee P f0ced467f93d42cdbb6c0a12a812b56a: Bootstrap complete.
I20240503 01:59:06.959501 4497 ts_tablet_manager.cc:1367] T c0bce60681a34a96a83284c762d6feee P f0ced467f93d42cdbb6c0a12a812b56a: Time spent bootstrapping tablet: real 0.021s user 0.020s sys 0.000s
I20240503 01:59:06.960160 4497 raft_consensus.cc:348] T c0bce60681a34a96a83284c762d6feee P f0ced467f93d42cdbb6c0a12a812b56a [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:59:06.960415 4497 raft_consensus.cc:729] T c0bce60681a34a96a83284c762d6feee P f0ced467f93d42cdbb6c0a12a812b56a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0ced467f93d42cdbb6c0a12a812b56a, State: Initialized, Role: FOLLOWER
I20240503 01:59:06.960528 4497 consensus_queue.cc:260] T c0bce60681a34a96a83284c762d6feee P f0ced467f93d42cdbb6c0a12a812b56a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "f0ced467f93d42cdbb6c0a12a812b56a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33129 } }
I20240503 01:59:06.960917 4497 ts_tablet_manager.cc:1398] T c0bce60681a34a96a83284c762d6feee P f0ced467f93d42cdbb6c0a12a812b56a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:06.961627 2419 tablet_copy_service.cc:342] P 80a019d9d1c34ccd9487207f90e68ab0: Request end of tablet copy session f0ced467f93d42cdbb6c0a12a812b56a-c0bce60681a34a96a83284c762d6feee received from {username='slave'} at 127.0.99.134:53317
I20240503 01:59:06.961769 2419 tablet_copy_service.cc:433] P 80a019d9d1c34ccd9487207f90e68ab0: ending tablet copy session f0ced467f93d42cdbb6c0a12a812b56a-c0bce60681a34a96a83284c762d6feee on tablet c0bce60681a34a96a83284c762d6feee with peer f0ced467f93d42cdbb6c0a12a812b56a
I20240503 01:59:07.496933 3927 raft_consensus.cc:1203] T c0bce60681a34a96a83284c762d6feee P f0ced467f93d42cdbb6c0a12a812b56a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:07.498010 4505 raft_consensus.cc:2932] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0ced467f93d42cdbb6c0a12a812b56a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "f0ced467f93d42cdbb6c0a12a812b56a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33129 } } }
I20240503 01:59:07.498459 3927 raft_consensus.cc:2932] T c0bce60681a34a96a83284c762d6feee P f0ced467f93d42cdbb6c0a12a812b56a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER f0ced467f93d42cdbb6c0a12a812b56a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "f0ced467f93d42cdbb6c0a12a812b56a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33129 } } }
I20240503 01:59:07.499619 4505 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:43314 (request call id 0) took 1080 ms. Trace:
I20240503 01:59:07.499693 4505 rpcz_store.cc:276] 0503 01:59:06.419306 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 01:59:06.419391 (+ 85us) service_pool.cc:225] Handling call
0503 01:59:06.420155 (+ 764us) log.cc:842] Serialized 257 byte log entry
0503 01:59:07.499600 (+1079445us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":67,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":115,"thread_start_us":83,"threads_started":1,"wal-append.queue_time_us":158,"wal-append.run_cpu_time_us":346,"wal-append.run_wall_time_us":1000424}
I20240503 01:59:07.501348 2252 catalog_manager.cc:5484] T c0bce60681a34a96a83284c762d6feee P f0ced467f93d42cdbb6c0a12a812b56a reported cstate change: config changed from index -1 to 4, VOTER f0ced467f93d42cdbb6c0a12a812b56a (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "80a019d9d1c34ccd9487207f90e68ab0" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "f0ced467f93d42cdbb6c0a12a812b56a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33129 } } }
I20240503 01:59:07.581964 3907 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:07.582067 3643 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:07.587052 3776 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:07.590571 3514 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:07.596061 2510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:07.600680 4039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:07.607452 2379 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:07.636694 2399 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49706355c8864e638e187eee7711cac3"
dest_uuid: "80a019d9d1c34ccd9487207f90e68ab0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43350
I20240503 01:59:07.636883 2399 raft_consensus.cc:595] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:07.698972 3643 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:07.699074 3776 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:07.702270 3514 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:07.703490 2510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:07.703765 3907 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:07.712774 2379 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:07.727584 4039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:07.812395 4039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:07.813443 2379 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:07.819308 3907 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:07.820261 2510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:07.825055 3514 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:07.833889 3643 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:07.845492 3776 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:07.876749 2530 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "986d142f7be641ed9a9c338c41ba430a"
dest_uuid: "86ed23734e434f8784885ebf9f1423b2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52978
I20240503 01:59:07.877045 2530 raft_consensus.cc:595] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:07.938167 4039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:07.938891 3776 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:07.940429 3643 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:07.948303 3514 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:07.958046 3907 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:07.958456 2510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:07.965353 2379 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:08.010320 4505 raft_consensus.cc:979] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0: : Instructing follower 03ea675b68a74c1192fa1a96f89a494b to start an election
I20240503 01:59:08.010458 4505 raft_consensus.cc:1067] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Signalling peer 03ea675b68a74c1192fa1a96f89a494b to start an election
I20240503 01:59:08.011265 3796 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "49706355c8864e638e187eee7711cac3"
dest_uuid: "03ea675b68a74c1192fa1a96f89a494b"
from {username='slave'} at 127.0.99.129:33699
I20240503 01:59:08.011530 3796 raft_consensus.cc:482] T 49706355c8864e638e187eee7711cac3 P 03ea675b68a74c1192fa1a96f89a494b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:08.011654 3796 raft_consensus.cc:3037] T 49706355c8864e638e187eee7711cac3 P 03ea675b68a74c1192fa1a96f89a494b [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:08.061367 2379 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:08.061717 4039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:08.065164 2510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:08.078773 3796 raft_consensus.cc:504] T 49706355c8864e638e187eee7711cac3 P 03ea675b68a74c1192fa1a96f89a494b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "03ea675b68a74c1192fa1a96f89a494b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38459 } }
I20240503 01:59:08.079849 3796 leader_election.cc:288] T 49706355c8864e638e187eee7711cac3 P 03ea675b68a74c1192fa1a96f89a494b [CANDIDATE]: Term 2 election: Requested vote from peers 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129:46305)
I20240503 01:59:08.081988 3514 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:08.084770 2399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49706355c8864e638e187eee7711cac3" candidate_uuid: "03ea675b68a74c1192fa1a96f89a494b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "80a019d9d1c34ccd9487207f90e68ab0"
I20240503 01:59:08.084973 2399 raft_consensus.cc:3032] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:08.085022 2399 raft_consensus.cc:729] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 80a019d9d1c34ccd9487207f90e68ab0, State: Running, Role: LEADER
I20240503 01:59:08.085165 2399 consensus_queue.cc:260] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "03ea675b68a74c1192fa1a96f89a494b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38459 } }
I20240503 01:59:08.085419 2399 raft_consensus.cc:3037] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:08.091723 3776 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:08.092486 3907 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:08.093420 3643 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:08.123391 2398 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c0bce60681a34a96a83284c762d6feee"
dest_uuid: "80a019d9d1c34ccd9487207f90e68ab0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43404
I20240503 01:59:08.123567 2398 raft_consensus.cc:595] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:08.132664 2399 raft_consensus.cc:2454] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03ea675b68a74c1192fa1a96f89a494b in term 2.
I20240503 01:59:08.133433 3730 leader_election.cc:302] T 49706355c8864e638e187eee7711cac3 P 03ea675b68a74c1192fa1a96f89a494b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 03ea675b68a74c1192fa1a96f89a494b, 80a019d9d1c34ccd9487207f90e68ab0; no voters:
I20240503 01:59:08.134603 4717 raft_consensus.cc:2781] T 49706355c8864e638e187eee7711cac3 P 03ea675b68a74c1192fa1a96f89a494b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:08.134816 4717 raft_consensus.cc:686] T 49706355c8864e638e187eee7711cac3 P 03ea675b68a74c1192fa1a96f89a494b [term 2 LEADER]: Becoming Leader. State: Replica: 03ea675b68a74c1192fa1a96f89a494b, State: Running, Role: LEADER
I20240503 01:59:08.134950 4679 raft_consensus.cc:979] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0: : Instructing follower f0ced467f93d42cdbb6c0a12a812b56a to start an election
I20240503 01:59:08.135015 4679 raft_consensus.cc:1067] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Signalling peer f0ced467f93d42cdbb6c0a12a812b56a to start an election
I20240503 01:59:08.135141 4717 consensus_queue.cc:237] T 49706355c8864e638e187eee7711cac3 P 03ea675b68a74c1192fa1a96f89a494b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "03ea675b68a74c1192fa1a96f89a494b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38459 } }
I20240503 01:59:08.135622 3927 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c0bce60681a34a96a83284c762d6feee"
dest_uuid: "f0ced467f93d42cdbb6c0a12a812b56a"
from {username='slave'} at 127.0.99.129:34047
I20240503 01:59:08.135803 3927 raft_consensus.cc:482] T c0bce60681a34a96a83284c762d6feee P f0ced467f93d42cdbb6c0a12a812b56a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:08.135907 3927 raft_consensus.cc:3037] T c0bce60681a34a96a83284c762d6feee P f0ced467f93d42cdbb6c0a12a812b56a [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:08.136847 3927 raft_consensus.cc:504] T c0bce60681a34a96a83284c762d6feee P f0ced467f93d42cdbb6c0a12a812b56a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "f0ced467f93d42cdbb6c0a12a812b56a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33129 } }
I20240503 01:59:08.137652 3927 leader_election.cc:288] T c0bce60681a34a96a83284c762d6feee P f0ced467f93d42cdbb6c0a12a812b56a [CANDIDATE]: Term 2 election: Requested vote from peers 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129:46305)
I20240503 01:59:08.138072 2399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0bce60681a34a96a83284c762d6feee" candidate_uuid: "f0ced467f93d42cdbb6c0a12a812b56a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "80a019d9d1c34ccd9487207f90e68ab0"
I20240503 01:59:08.138209 2399 raft_consensus.cc:3032] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:08.138254 2399 raft_consensus.cc:729] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 80a019d9d1c34ccd9487207f90e68ab0, State: Running, Role: LEADER
I20240503 01:59:08.138411 2399 consensus_queue.cc:260] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "f0ced467f93d42cdbb6c0a12a812b56a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33129 } }
I20240503 01:59:08.138581 2399 raft_consensus.cc:3037] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:08.139329 2255 catalog_manager.cc:5484] T 49706355c8864e638e187eee7711cac3 P 03ea675b68a74c1192fa1a96f89a494b reported cstate change: term changed from 1 to 2, leader changed from 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129) to 03ea675b68a74c1192fa1a96f89a494b (127.0.99.133). New cstate: current_term: 2 leader_uuid: "03ea675b68a74c1192fa1a96f89a494b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "03ea675b68a74c1192fa1a96f89a494b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38459 } } }
I20240503 01:59:08.139549 2399 raft_consensus.cc:2454] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0ced467f93d42cdbb6c0a12a812b56a in term 2.
I20240503 01:59:08.139910 3861 leader_election.cc:302] T c0bce60681a34a96a83284c762d6feee P f0ced467f93d42cdbb6c0a12a812b56a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 80a019d9d1c34ccd9487207f90e68ab0, f0ced467f93d42cdbb6c0a12a812b56a; no voters:
I20240503 01:59:08.140259 4721 raft_consensus.cc:2781] T c0bce60681a34a96a83284c762d6feee P f0ced467f93d42cdbb6c0a12a812b56a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:08.140455 4721 raft_consensus.cc:686] T c0bce60681a34a96a83284c762d6feee P f0ced467f93d42cdbb6c0a12a812b56a [term 2 LEADER]: Becoming Leader. State: Replica: f0ced467f93d42cdbb6c0a12a812b56a, State: Running, Role: LEADER
I20240503 01:59:08.140794 4721 consensus_queue.cc:237] T c0bce60681a34a96a83284c762d6feee P f0ced467f93d42cdbb6c0a12a812b56a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "f0ced467f93d42cdbb6c0a12a812b56a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33129 } }
I20240503 01:59:08.142745 2255 catalog_manager.cc:5484] T c0bce60681a34a96a83284c762d6feee P f0ced467f93d42cdbb6c0a12a812b56a reported cstate change: term changed from 1 to 2, leader changed from 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129) to f0ced467f93d42cdbb6c0a12a812b56a (127.0.99.134). New cstate: current_term: 2 leader_uuid: "f0ced467f93d42cdbb6c0a12a812b56a" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "f0ced467f93d42cdbb6c0a12a812b56a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33129 } } }
I20240503 01:59:08.186988 3776 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:08.187978 2379 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:08.189858 3907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:08.190819 4039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:08.193439 3643 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:08.213626 3514 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:08.214483 2510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:08.216500 4372 raft_consensus.cc:979] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2: : Instructing follower 65d5a1053fb5445486433fe66d82bdbe to start an election
I20240503 01:59:08.216601 4372 raft_consensus.cc:1067] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Signalling peer 65d5a1053fb5445486433fe66d82bdbe to start an election
I20240503 01:59:08.217324 3665 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "986d142f7be641ed9a9c338c41ba430a"
dest_uuid: "65d5a1053fb5445486433fe66d82bdbe"
from {username='slave'} at 127.0.99.130:41215
I20240503 01:59:08.217546 3665 raft_consensus.cc:482] T 986d142f7be641ed9a9c338c41ba430a P 65d5a1053fb5445486433fe66d82bdbe [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:08.217653 3665 raft_consensus.cc:3037] T 986d142f7be641ed9a9c338c41ba430a P 65d5a1053fb5445486433fe66d82bdbe [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:08.219234 3665 raft_consensus.cc:504] T 986d142f7be641ed9a9c338c41ba430a P 65d5a1053fb5445486433fe66d82bdbe [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "65d5a1053fb5445486433fe66d82bdbe" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43855 } }
I20240503 01:59:08.220979 3665 leader_election.cc:288] T 986d142f7be641ed9a9c338c41ba430a P 65d5a1053fb5445486433fe66d82bdbe [CANDIDATE]: Term 2 election: Requested vote from peers 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130:33335)
I20240503 01:59:08.221007 2530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986d142f7be641ed9a9c338c41ba430a" candidate_uuid: "65d5a1053fb5445486433fe66d82bdbe" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "86ed23734e434f8784885ebf9f1423b2"
I20240503 01:59:08.221210 2530 raft_consensus.cc:3032] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:08.221258 2530 raft_consensus.cc:729] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 86ed23734e434f8784885ebf9f1423b2, State: Running, Role: LEADER
I20240503 01:59:08.221417 2530 consensus_queue.cc:260] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "65d5a1053fb5445486433fe66d82bdbe" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43855 } }
I20240503 01:59:08.221668 2530 raft_consensus.cc:3037] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:08.222819 2530 raft_consensus.cc:2454] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65d5a1053fb5445486433fe66d82bdbe in term 2.
I20240503 01:59:08.223227 3599 leader_election.cc:302] T 986d142f7be641ed9a9c338c41ba430a P 65d5a1053fb5445486433fe66d82bdbe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 65d5a1053fb5445486433fe66d82bdbe, 86ed23734e434f8784885ebf9f1423b2; no voters:
I20240503 01:59:08.223901 4752 raft_consensus.cc:2781] T 986d142f7be641ed9a9c338c41ba430a P 65d5a1053fb5445486433fe66d82bdbe [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:08.224066 4752 raft_consensus.cc:686] T 986d142f7be641ed9a9c338c41ba430a P 65d5a1053fb5445486433fe66d82bdbe [term 2 LEADER]: Becoming Leader. State: Replica: 65d5a1053fb5445486433fe66d82bdbe, State: Running, Role: LEADER
I20240503 01:59:08.224359 4752 consensus_queue.cc:237] T 986d142f7be641ed9a9c338c41ba430a P 65d5a1053fb5445486433fe66d82bdbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "65d5a1053fb5445486433fe66d82bdbe" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43855 } }
I20240503 01:59:08.231415 2248 catalog_manager.cc:5484] T 986d142f7be641ed9a9c338c41ba430a P 65d5a1053fb5445486433fe66d82bdbe reported cstate change: term changed from 1 to 2, leader changed from 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130) to 65d5a1053fb5445486433fe66d82bdbe (127.0.99.132). New cstate: current_term: 2 leader_uuid: "65d5a1053fb5445486433fe66d82bdbe" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "65d5a1053fb5445486433fe66d82bdbe" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43855 } } }
I20240503 01:59:08.314635 2379 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:08.315939 2510 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:08.326839 3776 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:08.328873 3643 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:08.330776 3907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:08.330798 3514 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:08.338011 4039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:08.372679 2530 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "833d9ce859f84b499e667d86965d9fff"
dest_uuid: "86ed23734e434f8784885ebf9f1423b2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53026
I20240503 01:59:08.372850 2530 raft_consensus.cc:595] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:08.432797 2379 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:08.433545 4039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:08.434309 3776 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:08.435411 3643 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:08.436601 2510 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:08.447131 3514 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:08.463933 3907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:08.514645 4372 raft_consensus.cc:979] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2: : Instructing follower 5372f615183544f8bffc0fe13f4a152f to start an election
I20240503 01:59:08.514793 4372 raft_consensus.cc:1067] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Signalling peer 5372f615183544f8bffc0fe13f4a152f to start an election
I20240503 01:59:08.515480 4059 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "833d9ce859f84b499e667d86965d9fff"
dest_uuid: "5372f615183544f8bffc0fe13f4a152f"
from {username='slave'} at 127.0.99.130:51315
I20240503 01:59:08.515717 4059 raft_consensus.cc:482] T 833d9ce859f84b499e667d86965d9fff P 5372f615183544f8bffc0fe13f4a152f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:08.515830 4059 raft_consensus.cc:3037] T 833d9ce859f84b499e667d86965d9fff P 5372f615183544f8bffc0fe13f4a152f [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:08.516984 4059 raft_consensus.cc:504] T 833d9ce859f84b499e667d86965d9fff P 5372f615183544f8bffc0fe13f4a152f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "5372f615183544f8bffc0fe13f4a152f" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45035 } }
I20240503 01:59:08.518049 4059 leader_election.cc:288] T 833d9ce859f84b499e667d86965d9fff P 5372f615183544f8bffc0fe13f4a152f [CANDIDATE]: Term 2 election: Requested vote from peers 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130:33335)
I20240503 01:59:08.518575 2530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "833d9ce859f84b499e667d86965d9fff" candidate_uuid: "5372f615183544f8bffc0fe13f4a152f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "86ed23734e434f8784885ebf9f1423b2"
I20240503 01:59:08.518720 2530 raft_consensus.cc:3032] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:08.518774 2530 raft_consensus.cc:729] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 86ed23734e434f8784885ebf9f1423b2, State: Running, Role: LEADER
I20240503 01:59:08.518945 2530 consensus_queue.cc:260] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "5372f615183544f8bffc0fe13f4a152f" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45035 } }
I20240503 01:59:08.519107 2530 raft_consensus.cc:3037] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:08.519992 2530 raft_consensus.cc:2454] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5372f615183544f8bffc0fe13f4a152f in term 2.
I20240503 01:59:08.520354 3992 leader_election.cc:302] T 833d9ce859f84b499e667d86965d9fff P 5372f615183544f8bffc0fe13f4a152f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5372f615183544f8bffc0fe13f4a152f, 86ed23734e434f8784885ebf9f1423b2; no voters:
I20240503 01:59:08.520649 4844 raft_consensus.cc:2781] T 833d9ce859f84b499e667d86965d9fff P 5372f615183544f8bffc0fe13f4a152f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:08.520850 4844 raft_consensus.cc:686] T 833d9ce859f84b499e667d86965d9fff P 5372f615183544f8bffc0fe13f4a152f [term 2 LEADER]: Becoming Leader. State: Replica: 5372f615183544f8bffc0fe13f4a152f, State: Running, Role: LEADER
I20240503 01:59:08.521020 4844 consensus_queue.cc:237] T 833d9ce859f84b499e667d86965d9fff P 5372f615183544f8bffc0fe13f4a152f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "5372f615183544f8bffc0fe13f4a152f" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45035 } }
I20240503 01:59:08.522738 2252 catalog_manager.cc:5484] T 833d9ce859f84b499e667d86965d9fff P 5372f615183544f8bffc0fe13f4a152f reported cstate change: term changed from 1 to 2, leader changed from 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130) to 5372f615183544f8bffc0fe13f4a152f (127.0.99.135). New cstate: current_term: 2 leader_uuid: "5372f615183544f8bffc0fe13f4a152f" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "5372f615183544f8bffc0fe13f4a152f" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45035 } } }
I20240503 01:59:08.553769 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:08.555769 3776 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:08.556910 2379 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:08.563066 2399 raft_consensus.cc:1261] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 2 FOLLOWER]: Refusing update from remote peer 03ea675b68a74c1192fa1a96f89a494b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 01:59:08.563899 4717 consensus_queue.cc:1035] T 49706355c8864e638e187eee7711cac3 P 03ea675b68a74c1192fa1a96f89a494b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:08.568351 3643 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:08.570233 3514 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:08.570894 2510 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:08.574146 3907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:08.602599 2399 raft_consensus.cc:1261] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0 [term 2 FOLLOWER]: Refusing update from remote peer f0ced467f93d42cdbb6c0a12a812b56a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 01:59:08.603200 4721 consensus_queue.cc:1035] T c0bce60681a34a96a83284c762d6feee P f0ced467f93d42cdbb6c0a12a812b56a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:08.613950 2530 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a614380f92f64189ac5b5cdb42edf2e6"
dest_uuid: "86ed23734e434f8784885ebf9f1423b2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53050
I20240503 01:59:08.614094 2530 raft_consensus.cc:595] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:08.670459 2379 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:08.671831 2510 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:08.671885 3776 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:08.678383 2530 raft_consensus.cc:1261] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2 [term 2 FOLLOWER]: Refusing update from remote peer 65d5a1053fb5445486433fe66d82bdbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 01:59:08.678799 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:08.679620 4753 consensus_queue.cc:1035] T 986d142f7be641ed9a9c338c41ba430a P 65d5a1053fb5445486433fe66d82bdbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:08.687898 3514 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:08.695729 3907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:08.709511 3643 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:08.801512 3907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:08.807538 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:08.812443 2379 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:08.814785 2510 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:08.816438 3643 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:08.819164 3776 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:08.820783 3514 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:08.863497 2399 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "358379b5f00a41d69db831501091e702"
dest_uuid: "80a019d9d1c34ccd9487207f90e68ab0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43488
I20240503 01:59:08.863654 2399 raft_consensus.cc:595] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:08.922200 3776 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:08.922861 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:08.927495 3907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:08.934935 3643 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:08.935590 3514 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:08.938264 2510 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:08.940337 2379 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:08.996865 2530 raft_consensus.cc:1261] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2 [term 2 FOLLOWER]: Refusing update from remote peer 5372f615183544f8bffc0fe13f4a152f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 01:59:08.997478 4844 consensus_queue.cc:1035] T 833d9ce859f84b499e667d86965d9fff P 5372f615183544f8bffc0fe13f4a152f [LEADER]: Connected to new peer: Peer: permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:09.034487 4372 raft_consensus.cc:979] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2: : Instructing follower cc5fffa9b8844c6f9c948344d8081833 to start an election
I20240503 01:59:09.034631 4372 raft_consensus.cc:1067] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Signalling peer cc5fffa9b8844c6f9c948344d8081833 to start an election
I20240503 01:59:09.035230 3534 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a614380f92f64189ac5b5cdb42edf2e6"
dest_uuid: "cc5fffa9b8844c6f9c948344d8081833"
from {username='slave'} at 127.0.99.130:56133
I20240503 01:59:09.035440 3534 raft_consensus.cc:482] T a614380f92f64189ac5b5cdb42edf2e6 P cc5fffa9b8844c6f9c948344d8081833 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:09.035548 3534 raft_consensus.cc:3037] T a614380f92f64189ac5b5cdb42edf2e6 P cc5fffa9b8844c6f9c948344d8081833 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:09.037086 3534 raft_consensus.cc:504] T a614380f92f64189ac5b5cdb42edf2e6 P cc5fffa9b8844c6f9c948344d8081833 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "cc5fffa9b8844c6f9c948344d8081833" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45579 } }
I20240503 01:59:09.038417 3534 leader_election.cc:288] T a614380f92f64189ac5b5cdb42edf2e6 P cc5fffa9b8844c6f9c948344d8081833 [CANDIDATE]: Term 2 election: Requested vote from peers 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130:33335)
I20240503 01:59:09.038895 2530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a614380f92f64189ac5b5cdb42edf2e6" candidate_uuid: "cc5fffa9b8844c6f9c948344d8081833" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "86ed23734e434f8784885ebf9f1423b2"
I20240503 01:59:09.039063 2530 raft_consensus.cc:3032] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:09.039119 2530 raft_consensus.cc:729] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 86ed23734e434f8784885ebf9f1423b2, State: Running, Role: LEADER
I20240503 01:59:09.039268 2530 consensus_queue.cc:260] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "cc5fffa9b8844c6f9c948344d8081833" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45579 } }
I20240503 01:59:09.039379 2530 raft_consensus.cc:3037] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:09.040532 2530 raft_consensus.cc:2454] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc5fffa9b8844c6f9c948344d8081833 in term 2.
I20240503 01:59:09.041126 3468 leader_election.cc:302] T a614380f92f64189ac5b5cdb42edf2e6 P cc5fffa9b8844c6f9c948344d8081833 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 86ed23734e434f8784885ebf9f1423b2, cc5fffa9b8844c6f9c948344d8081833; no voters:
I20240503 01:59:09.041970 5028 raft_consensus.cc:2781] T a614380f92f64189ac5b5cdb42edf2e6 P cc5fffa9b8844c6f9c948344d8081833 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:09.042182 5028 raft_consensus.cc:686] T a614380f92f64189ac5b5cdb42edf2e6 P cc5fffa9b8844c6f9c948344d8081833 [term 2 LEADER]: Becoming Leader. State: Replica: cc5fffa9b8844c6f9c948344d8081833, State: Running, Role: LEADER
I20240503 01:59:09.042590 5028 consensus_queue.cc:237] T a614380f92f64189ac5b5cdb42edf2e6 P cc5fffa9b8844c6f9c948344d8081833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "cc5fffa9b8844c6f9c948344d8081833" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45579 } }
I20240503 01:59:09.049854 2253 catalog_manager.cc:5484] T a614380f92f64189ac5b5cdb42edf2e6 P cc5fffa9b8844c6f9c948344d8081833 reported cstate change: term changed from 1 to 2, leader changed from 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130) to cc5fffa9b8844c6f9c948344d8081833 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "cc5fffa9b8844c6f9c948344d8081833" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "cc5fffa9b8844c6f9c948344d8081833" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45579 } } }
I20240503 01:59:09.051369 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:09.052330 3643 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:09.053614 3776 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:09.053941 2379 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:09.058909 2510 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:09.067634 3907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:09.081724 3514 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:09.114147 2399 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "32f685df61f144628bbb19c89e36ee6e"
dest_uuid: "80a019d9d1c34ccd9487207f90e68ab0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43518
I20240503 01:59:09.114297 2399 raft_consensus.cc:595] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:09.174198 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:09.175407 3776 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:09.175695 2379 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:09.179728 3514 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:09.179908 2510 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:09.180696 3643 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:09.199340 3907 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:09.251857 4505 raft_consensus.cc:979] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0: : Instructing follower f0ced467f93d42cdbb6c0a12a812b56a to start an election
I20240503 01:59:09.252048 4505 raft_consensus.cc:1067] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Signalling peer f0ced467f93d42cdbb6c0a12a812b56a to start an election
I20240503 01:59:09.252487 3927 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "32f685df61f144628bbb19c89e36ee6e"
dest_uuid: "f0ced467f93d42cdbb6c0a12a812b56a"
from {username='slave'} at 127.0.99.129:34047
I20240503 01:59:09.252633 3927 raft_consensus.cc:482] T 32f685df61f144628bbb19c89e36ee6e P f0ced467f93d42cdbb6c0a12a812b56a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:09.252703 3927 raft_consensus.cc:3037] T 32f685df61f144628bbb19c89e36ee6e P f0ced467f93d42cdbb6c0a12a812b56a [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:09.253708 3927 raft_consensus.cc:504] T 32f685df61f144628bbb19c89e36ee6e P f0ced467f93d42cdbb6c0a12a812b56a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "f0ced467f93d42cdbb6c0a12a812b56a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33129 } }
I20240503 01:59:09.254067 3927 leader_election.cc:288] T 32f685df61f144628bbb19c89e36ee6e P f0ced467f93d42cdbb6c0a12a812b56a [CANDIDATE]: Term 2 election: Requested vote from peers 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129:46305)
I20240503 01:59:09.254561 2399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f685df61f144628bbb19c89e36ee6e" candidate_uuid: "f0ced467f93d42cdbb6c0a12a812b56a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "80a019d9d1c34ccd9487207f90e68ab0"
I20240503 01:59:09.254701 2399 raft_consensus.cc:3032] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:09.254751 2399 raft_consensus.cc:729] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 80a019d9d1c34ccd9487207f90e68ab0, State: Running, Role: LEADER
I20240503 01:59:09.254942 2399 consensus_queue.cc:260] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "f0ced467f93d42cdbb6c0a12a812b56a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33129 } }
I20240503 01:59:09.255105 2399 raft_consensus.cc:3037] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:09.256124 2399 raft_consensus.cc:2454] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0ced467f93d42cdbb6c0a12a812b56a in term 2.
I20240503 01:59:09.256420 3861 leader_election.cc:302] T 32f685df61f144628bbb19c89e36ee6e P f0ced467f93d42cdbb6c0a12a812b56a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 80a019d9d1c34ccd9487207f90e68ab0, f0ced467f93d42cdbb6c0a12a812b56a; no voters:
I20240503 01:59:09.256603 4722 raft_consensus.cc:2781] T 32f685df61f144628bbb19c89e36ee6e P f0ced467f93d42cdbb6c0a12a812b56a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:09.256767 4722 raft_consensus.cc:686] T 32f685df61f144628bbb19c89e36ee6e P f0ced467f93d42cdbb6c0a12a812b56a [term 2 LEADER]: Becoming Leader. State: Replica: f0ced467f93d42cdbb6c0a12a812b56a, State: Running, Role: LEADER
I20240503 01:59:09.256904 4722 consensus_queue.cc:237] T 32f685df61f144628bbb19c89e36ee6e P f0ced467f93d42cdbb6c0a12a812b56a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "f0ced467f93d42cdbb6c0a12a812b56a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33129 } }
I20240503 01:59:09.258268 2248 catalog_manager.cc:5484] T 32f685df61f144628bbb19c89e36ee6e P f0ced467f93d42cdbb6c0a12a812b56a reported cstate change: term changed from 1 to 2, leader changed from 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129) to f0ced467f93d42cdbb6c0a12a812b56a (127.0.99.134). New cstate: current_term: 2 leader_uuid: "f0ced467f93d42cdbb6c0a12a812b56a" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "f0ced467f93d42cdbb6c0a12a812b56a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33129 } } }
I20240503 01:59:09.287662 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:09.288266 2379 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:09.290293 3776 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:09.302069 2510 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:09.302340 3643 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:09.303666 3514 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:09.315439 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:09.348536 2530 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5c289e9ad385459aa89fa32637a42f36"
dest_uuid: "86ed23734e434f8784885ebf9f1423b2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53122
I20240503 01:59:09.348690 2530 raft_consensus.cc:595] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:09.379853 4505 raft_consensus.cc:979] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0: : Instructing follower 03ea675b68a74c1192fa1a96f89a494b to start an election
I20240503 01:59:09.380013 4505 raft_consensus.cc:1067] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Signalling peer 03ea675b68a74c1192fa1a96f89a494b to start an election
I20240503 01:59:09.381163 3796 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "358379b5f00a41d69db831501091e702"
dest_uuid: "03ea675b68a74c1192fa1a96f89a494b"
from {username='slave'} at 127.0.99.129:33699
I20240503 01:59:09.381315 3796 raft_consensus.cc:482] T 358379b5f00a41d69db831501091e702 P 03ea675b68a74c1192fa1a96f89a494b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:09.381387 3796 raft_consensus.cc:3037] T 358379b5f00a41d69db831501091e702 P 03ea675b68a74c1192fa1a96f89a494b [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:09.382472 3796 raft_consensus.cc:504] T 358379b5f00a41d69db831501091e702 P 03ea675b68a74c1192fa1a96f89a494b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "03ea675b68a74c1192fa1a96f89a494b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38459 } }
I20240503 01:59:09.382854 3796 leader_election.cc:288] T 358379b5f00a41d69db831501091e702 P 03ea675b68a74c1192fa1a96f89a494b [CANDIDATE]: Term 2 election: Requested vote from peers 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129:46305)
I20240503 01:59:09.383414 2399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358379b5f00a41d69db831501091e702" candidate_uuid: "03ea675b68a74c1192fa1a96f89a494b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "80a019d9d1c34ccd9487207f90e68ab0"
I20240503 01:59:09.383550 2399 raft_consensus.cc:3032] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:09.383647 2399 raft_consensus.cc:729] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 80a019d9d1c34ccd9487207f90e68ab0, State: Running, Role: LEADER
I20240503 01:59:09.383826 2399 consensus_queue.cc:260] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "03ea675b68a74c1192fa1a96f89a494b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38459 } }
I20240503 01:59:09.383993 2399 raft_consensus.cc:3037] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:09.385059 2399 raft_consensus.cc:2454] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03ea675b68a74c1192fa1a96f89a494b in term 2.
I20240503 01:59:09.385421 3730 leader_election.cc:302] T 358379b5f00a41d69db831501091e702 P 03ea675b68a74c1192fa1a96f89a494b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 03ea675b68a74c1192fa1a96f89a494b, 80a019d9d1c34ccd9487207f90e68ab0; no voters:
I20240503 01:59:09.385668 4717 raft_consensus.cc:2781] T 358379b5f00a41d69db831501091e702 P 03ea675b68a74c1192fa1a96f89a494b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:09.385850 4717 raft_consensus.cc:686] T 358379b5f00a41d69db831501091e702 P 03ea675b68a74c1192fa1a96f89a494b [term 2 LEADER]: Becoming Leader. State: Replica: 03ea675b68a74c1192fa1a96f89a494b, State: Running, Role: LEADER
I20240503 01:59:09.386031 4717 consensus_queue.cc:237] T 358379b5f00a41d69db831501091e702 P 03ea675b68a74c1192fa1a96f89a494b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "03ea675b68a74c1192fa1a96f89a494b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38459 } }
I20240503 01:59:09.387890 2252 catalog_manager.cc:5484] T 358379b5f00a41d69db831501091e702 P 03ea675b68a74c1192fa1a96f89a494b reported cstate change: term changed from 1 to 2, leader changed from 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129) to 03ea675b68a74c1192fa1a96f89a494b (127.0.99.133). New cstate: current_term: 2 leader_uuid: "03ea675b68a74c1192fa1a96f89a494b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "03ea675b68a74c1192fa1a96f89a494b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38459 } } }
I20240503 01:59:09.411623 2379 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:09.412441 3776 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:09.414155 2510 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:09.415103 3643 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:09.415148 3514 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:09.431034 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:09.440855 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:09.491663 2530 raft_consensus.cc:1261] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2 [term 2 FOLLOWER]: Refusing update from remote peer cc5fffa9b8844c6f9c948344d8081833: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 01:59:09.492254 5030 consensus_queue.cc:1035] T a614380f92f64189ac5b5cdb42edf2e6 P cc5fffa9b8844c6f9c948344d8081833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:09.527460 3776 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:09.529269 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:09.533911 2379 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:09.536882 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:09.543349 3643 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:09.543656 3514 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:09.554977 2510 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:09.591154 2399 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ae344bce9aeb452b848accc96ddbb1ac"
dest_uuid: "80a019d9d1c34ccd9487207f90e68ab0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43556
I20240503 01:59:09.591308 2399 raft_consensus.cc:595] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:09.660403 3776 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:09.660496 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:09.661518 3643 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:09.667546 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:09.668733 3514 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:09.680081 2510 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:09.687605 5018 raft_consensus.cc:979] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2: : Instructing follower cc5fffa9b8844c6f9c948344d8081833 to start an election
I20240503 01:59:09.687724 5018 raft_consensus.cc:1067] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Signalling peer cc5fffa9b8844c6f9c948344d8081833 to start an election
I20240503 01:59:09.688028 2379 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:09.688119 3534 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5c289e9ad385459aa89fa32637a42f36"
dest_uuid: "cc5fffa9b8844c6f9c948344d8081833"
from {username='slave'} at 127.0.99.130:56133
I20240503 01:59:09.688257 3534 raft_consensus.cc:482] T 5c289e9ad385459aa89fa32637a42f36 P cc5fffa9b8844c6f9c948344d8081833 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:09.688304 3534 raft_consensus.cc:3037] T 5c289e9ad385459aa89fa32637a42f36 P cc5fffa9b8844c6f9c948344d8081833 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:09.689086 3534 raft_consensus.cc:504] T 5c289e9ad385459aa89fa32637a42f36 P cc5fffa9b8844c6f9c948344d8081833 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "cc5fffa9b8844c6f9c948344d8081833" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45579 } }
I20240503 01:59:09.689344 3534 leader_election.cc:288] T 5c289e9ad385459aa89fa32637a42f36 P cc5fffa9b8844c6f9c948344d8081833 [CANDIDATE]: Term 2 election: Requested vote from peers 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130:33335)
I20240503 01:59:09.689795 2530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c289e9ad385459aa89fa32637a42f36" candidate_uuid: "cc5fffa9b8844c6f9c948344d8081833" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "86ed23734e434f8784885ebf9f1423b2"
I20240503 01:59:09.689913 2530 raft_consensus.cc:3032] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:09.689958 2530 raft_consensus.cc:729] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 86ed23734e434f8784885ebf9f1423b2, State: Running, Role: LEADER
I20240503 01:59:09.690109 2530 consensus_queue.cc:260] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "cc5fffa9b8844c6f9c948344d8081833" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45579 } }
I20240503 01:59:09.690258 2530 raft_consensus.cc:3037] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:09.691431 2530 raft_consensus.cc:2454] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc5fffa9b8844c6f9c948344d8081833 in term 2.
I20240503 01:59:09.692332 3468 leader_election.cc:302] T 5c289e9ad385459aa89fa32637a42f36 P cc5fffa9b8844c6f9c948344d8081833 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 86ed23734e434f8784885ebf9f1423b2, cc5fffa9b8844c6f9c948344d8081833; no voters:
I20240503 01:59:09.692580 5028 raft_consensus.cc:2781] T 5c289e9ad385459aa89fa32637a42f36 P cc5fffa9b8844c6f9c948344d8081833 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:09.692662 5028 raft_consensus.cc:686] T 5c289e9ad385459aa89fa32637a42f36 P cc5fffa9b8844c6f9c948344d8081833 [term 2 LEADER]: Becoming Leader. State: Replica: cc5fffa9b8844c6f9c948344d8081833, State: Running, Role: LEADER
I20240503 01:59:09.692781 5028 consensus_queue.cc:237] T 5c289e9ad385459aa89fa32637a42f36 P cc5fffa9b8844c6f9c948344d8081833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "cc5fffa9b8844c6f9c948344d8081833" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45579 } }
I20240503 01:59:09.703359 2255 catalog_manager.cc:5484] T 5c289e9ad385459aa89fa32637a42f36 P cc5fffa9b8844c6f9c948344d8081833 reported cstate change: term changed from 1 to 2, leader changed from 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130) to cc5fffa9b8844c6f9c948344d8081833 (127.0.99.131). New cstate: current_term: 2 leader_uuid: "cc5fffa9b8844c6f9c948344d8081833" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "cc5fffa9b8844c6f9c948344d8081833" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45579 } } }
I20240503 01:59:09.757755 2399 raft_consensus.cc:1261] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0 [term 2 FOLLOWER]: Refusing update from remote peer f0ced467f93d42cdbb6c0a12a812b56a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 01:59:09.758272 4722 consensus_queue.cc:1035] T 32f685df61f144628bbb19c89e36ee6e P f0ced467f93d42cdbb6c0a12a812b56a [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:09.869266 5087 raft_consensus.cc:979] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0: : Instructing follower 65d5a1053fb5445486433fe66d82bdbe to start an election
I20240503 01:59:09.869398 5087 raft_consensus.cc:1067] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Signalling peer 65d5a1053fb5445486433fe66d82bdbe to start an election
I20240503 01:59:09.870011 3665 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae344bce9aeb452b848accc96ddbb1ac"
dest_uuid: "65d5a1053fb5445486433fe66d82bdbe"
from {username='slave'} at 127.0.99.129:39291
I20240503 01:59:09.870256 3665 raft_consensus.cc:482] T ae344bce9aeb452b848accc96ddbb1ac P 65d5a1053fb5445486433fe66d82bdbe [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:09.870328 3665 raft_consensus.cc:3037] T ae344bce9aeb452b848accc96ddbb1ac P 65d5a1053fb5445486433fe66d82bdbe [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:09.871243 3665 raft_consensus.cc:504] T ae344bce9aeb452b848accc96ddbb1ac P 65d5a1053fb5445486433fe66d82bdbe [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "65d5a1053fb5445486433fe66d82bdbe" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43855 } }
I20240503 01:59:09.871613 3665 leader_election.cc:288] T ae344bce9aeb452b848accc96ddbb1ac P 65d5a1053fb5445486433fe66d82bdbe [CANDIDATE]: Term 2 election: Requested vote from peers 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129:46305)
I20240503 01:59:09.872066 2399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae344bce9aeb452b848accc96ddbb1ac" candidate_uuid: "65d5a1053fb5445486433fe66d82bdbe" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "80a019d9d1c34ccd9487207f90e68ab0"
I20240503 01:59:09.872205 2399 raft_consensus.cc:3032] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:09.872257 2399 raft_consensus.cc:729] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 80a019d9d1c34ccd9487207f90e68ab0, State: Running, Role: LEADER
I20240503 01:59:09.872413 2399 consensus_queue.cc:260] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "65d5a1053fb5445486433fe66d82bdbe" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43855 } }
I20240503 01:59:09.872545 2399 raft_consensus.cc:3037] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:09.873399 2399 raft_consensus.cc:2454] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65d5a1053fb5445486433fe66d82bdbe in term 2.
I20240503 01:59:09.873736 3599 leader_election.cc:302] T ae344bce9aeb452b848accc96ddbb1ac P 65d5a1053fb5445486433fe66d82bdbe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 65d5a1053fb5445486433fe66d82bdbe, 80a019d9d1c34ccd9487207f90e68ab0; no voters:
I20240503 01:59:09.873978 5229 raft_consensus.cc:2781] T ae344bce9aeb452b848accc96ddbb1ac P 65d5a1053fb5445486433fe66d82bdbe [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:09.874178 5229 raft_consensus.cc:686] T ae344bce9aeb452b848accc96ddbb1ac P 65d5a1053fb5445486433fe66d82bdbe [term 2 LEADER]: Becoming Leader. State: Replica: 65d5a1053fb5445486433fe66d82bdbe, State: Running, Role: LEADER
I20240503 01:59:09.874320 5229 consensus_queue.cc:237] T ae344bce9aeb452b848accc96ddbb1ac P 65d5a1053fb5445486433fe66d82bdbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "65d5a1053fb5445486433fe66d82bdbe" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43855 } }
I20240503 01:59:09.876021 2255 catalog_manager.cc:5484] T ae344bce9aeb452b848accc96ddbb1ac P 65d5a1053fb5445486433fe66d82bdbe reported cstate change: term changed from 1 to 2, leader changed from 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129) to 65d5a1053fb5445486433fe66d82bdbe (127.0.99.132). New cstate: current_term: 2 leader_uuid: "65d5a1053fb5445486433fe66d82bdbe" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "65d5a1053fb5445486433fe66d82bdbe" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43855 } } }
I20240503 01:59:09.982933 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:09.983875 3776 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:09.984463 2379 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:09.984933 2510 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:09.987565 3643 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:09.992437 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:09.998709 2399 raft_consensus.cc:1261] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0 [term 2 FOLLOWER]: Refusing update from remote peer 03ea675b68a74c1192fa1a96f89a494b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 01:59:09.999231 4717 consensus_queue.cc:1035] T 358379b5f00a41d69db831501091e702 P 03ea675b68a74c1192fa1a96f89a494b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:10.013262 3514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.060199 3796 consensus_queue.cc:237] T 49706355c8864e638e187eee7711cac3 P 03ea675b68a74c1192fa1a96f89a494b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "03ea675b68a74c1192fa1a96f89a494b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38459 } }
I20240503 01:59:10.061220 5276 raft_consensus.cc:2932] T 49706355c8864e638e187eee7711cac3 P 03ea675b68a74c1192fa1a96f89a494b [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "03ea675b68a74c1192fa1a96f89a494b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38459 } } }
I20240503 01:59:10.063977 2255 catalog_manager.cc:5484] T 49706355c8864e638e187eee7711cac3 P 03ea675b68a74c1192fa1a96f89a494b reported cstate change: config changed from index 4 to 6, VOTER 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "03ea675b68a74c1192fa1a96f89a494b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "03ea675b68a74c1192fa1a96f89a494b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38459 } } }
I20240503 01:59:10.067092 2379 tablet_service.cc:1514] Processing DeleteTablet for tablet 49706355c8864e638e187eee7711cac3 with delete_type TABLET_DATA_TOMBSTONED (TS 80a019d9d1c34ccd9487207f90e68ab0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47770
I20240503 01:59:10.067620 5299 tablet_replica.cc:330] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0: stopping tablet replica
I20240503 01:59:10.067930 5299 raft_consensus.cc:2229] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:10.068118 5299 raft_consensus.cc:2258] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:10.069106 5299 ts_tablet_manager.cc:1875] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:10.071177 5299 ts_tablet_manager.cc:1888] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 01:59:10.071267 5299 log.cc:1196] T 49706355c8864e638e187eee7711cac3 P 80a019d9d1c34ccd9487207f90e68ab0: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/49706355c8864e638e187eee7711cac3
I20240503 01:59:10.071924 2242 catalog_manager.cc:4830] TS 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129:46305): tablet 49706355c8864e638e187eee7711cac3 (table rebalance_test_table_3 [id=cc6ae6794e364638a2b3a21c6e7329ea]) successfully deleted
I20240503 01:59:10.137662 3776 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.138240 2379 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:10.139057 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.143182 2510 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:10.145889 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.153435 3643 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.160123 3514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.212728 3665 consensus_queue.cc:237] T 986d142f7be641ed9a9c338c41ba430a P 65d5a1053fb5445486433fe66d82bdbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "65d5a1053fb5445486433fe66d82bdbe" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43855 } }
I20240503 01:59:10.213718 5229 raft_consensus.cc:2932] T 986d142f7be641ed9a9c338c41ba430a P 65d5a1053fb5445486433fe66d82bdbe [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "65d5a1053fb5445486433fe66d82bdbe" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43855 } } }
I20240503 01:59:10.216341 2257 catalog_manager.cc:5484] T 986d142f7be641ed9a9c338c41ba430a P 65d5a1053fb5445486433fe66d82bdbe reported cstate change: config changed from index 4 to 6, VOTER 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "65d5a1053fb5445486433fe66d82bdbe" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "65d5a1053fb5445486433fe66d82bdbe" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43855 } } }
I20240503 01:59:10.219383 2510 tablet_service.cc:1514] Processing DeleteTablet for tablet 986d142f7be641ed9a9c338c41ba430a with delete_type TABLET_DATA_TOMBSTONED (TS 86ed23734e434f8784885ebf9f1423b2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44718
I20240503 01:59:10.219883 5353 tablet_replica.cc:330] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2: stopping tablet replica
I20240503 01:59:10.220170 5353 raft_consensus.cc:2229] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:10.220332 5353 raft_consensus.cc:2258] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:10.221205 5353 ts_tablet_manager.cc:1875] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:10.223318 5353 ts_tablet_manager.cc:1888] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 01:59:10.223425 5353 log.cc:1196] T 986d142f7be641ed9a9c338c41ba430a P 86ed23734e434f8784885ebf9f1423b2: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/986d142f7be641ed9a9c338c41ba430a
I20240503 01:59:10.224036 2242 catalog_manager.cc:4830] TS 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130:33335): tablet 986d142f7be641ed9a9c338c41ba430a (table rebalance_test_table_3 [id=cc6ae6794e364638a2b3a21c6e7329ea]) successfully deleted
I20240503 01:59:10.259119 2530 raft_consensus.cc:1261] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2 [term 2 FOLLOWER]: Refusing update from remote peer cc5fffa9b8844c6f9c948344d8081833: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 01:59:10.259617 5030 consensus_queue.cc:1035] T 5c289e9ad385459aa89fa32637a42f36 P cc5fffa9b8844c6f9c948344d8081833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:10.284322 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.285545 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.288015 2379 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:10.289134 3776 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.299816 2399 raft_consensus.cc:1261] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0 [term 2 FOLLOWER]: Refusing update from remote peer 65d5a1053fb5445486433fe66d82bdbe: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 01:59:10.300437 5236 consensus_queue.cc:1035] T ae344bce9aeb452b848accc96ddbb1ac P 65d5a1053fb5445486433fe66d82bdbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:10.300643 2510 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:10.300966 3514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.311164 3643 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.358145 3927 consensus_queue.cc:237] T c0bce60681a34a96a83284c762d6feee P f0ced467f93d42cdbb6c0a12a812b56a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f0ced467f93d42cdbb6c0a12a812b56a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33129 } }
I20240503 01:59:10.359047 5233 raft_consensus.cc:2932] T c0bce60681a34a96a83284c762d6feee P f0ced467f93d42cdbb6c0a12a812b56a [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f0ced467f93d42cdbb6c0a12a812b56a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33129 } } }
I20240503 01:59:10.362113 2256 catalog_manager.cc:5484] T c0bce60681a34a96a83284c762d6feee P f0ced467f93d42cdbb6c0a12a812b56a reported cstate change: config changed from index 4 to 6, VOTER 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "f0ced467f93d42cdbb6c0a12a812b56a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f0ced467f93d42cdbb6c0a12a812b56a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33129 } } }
I20240503 01:59:10.365643 2379 tablet_service.cc:1514] Processing DeleteTablet for tablet c0bce60681a34a96a83284c762d6feee with delete_type TABLET_DATA_TOMBSTONED (TS 80a019d9d1c34ccd9487207f90e68ab0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47770
I20240503 01:59:10.365957 5299 tablet_replica.cc:330] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0: stopping tablet replica
I20240503 01:59:10.366106 5299 raft_consensus.cc:2229] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:10.366225 5299 raft_consensus.cc:2258] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:10.366780 5299 ts_tablet_manager.cc:1875] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:10.368731 5299 ts_tablet_manager.cc:1888] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 01:59:10.368808 5299 log.cc:1196] T c0bce60681a34a96a83284c762d6feee P 80a019d9d1c34ccd9487207f90e68ab0: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/c0bce60681a34a96a83284c762d6feee
I20240503 01:59:10.369379 2242 catalog_manager.cc:4830] TS 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129:46305): tablet c0bce60681a34a96a83284c762d6feee (table rebalance_test_table_4 [id=e3217159b04d4bebadaaac3cc7d7cec1]) successfully deleted
I20240503 01:59:10.429109 2379 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:10.429157 2510 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:10.429845 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.431411 3776 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.431865 3514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.442745 3643 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.452608 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.495693 4059 consensus_queue.cc:237] T 833d9ce859f84b499e667d86965d9fff P 5372f615183544f8bffc0fe13f4a152f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5372f615183544f8bffc0fe13f4a152f" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45035 } }
I20240503 01:59:10.496565 5464 raft_consensus.cc:2932] T 833d9ce859f84b499e667d86965d9fff P 5372f615183544f8bffc0fe13f4a152f [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5372f615183544f8bffc0fe13f4a152f" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45035 } } }
I20240503 01:59:10.499049 2248 catalog_manager.cc:5484] T 833d9ce859f84b499e667d86965d9fff P 5372f615183544f8bffc0fe13f4a152f reported cstate change: config changed from index 4 to 6, VOTER 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "5372f615183544f8bffc0fe13f4a152f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5372f615183544f8bffc0fe13f4a152f" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 45035 } } }
I20240503 01:59:10.501703 2510 tablet_service.cc:1514] Processing DeleteTablet for tablet 833d9ce859f84b499e667d86965d9fff with delete_type TABLET_DATA_TOMBSTONED (TS 86ed23734e434f8784885ebf9f1423b2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44718
I20240503 01:59:10.501955 5353 tablet_replica.cc:330] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2: stopping tablet replica
I20240503 01:59:10.502094 5353 raft_consensus.cc:2229] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:10.502209 5353 raft_consensus.cc:2258] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:10.502854 5353 ts_tablet_manager.cc:1875] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:10.504707 5353 ts_tablet_manager.cc:1888] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 01:59:10.504781 5353 log.cc:1196] T 833d9ce859f84b499e667d86965d9fff P 86ed23734e434f8784885ebf9f1423b2: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/833d9ce859f84b499e667d86965d9fff
I20240503 01:59:10.505479 2242 catalog_manager.cc:4830] TS 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130:33335): tablet 833d9ce859f84b499e667d86965d9fff (table rebalance_test_table_2 [id=5e9078a735bc4854ab7fe5df318eca11]) successfully deleted
I20240503 01:59:10.561199 3776 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.562549 3643 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.567559 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.568068 2379 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:10.568147 2510 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:10.570317 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.571249 3514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.643328 3534 consensus_queue.cc:237] T a614380f92f64189ac5b5cdb42edf2e6 P cc5fffa9b8844c6f9c948344d8081833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cc5fffa9b8844c6f9c948344d8081833" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45579 } }
I20240503 01:59:10.644176 5030 raft_consensus.cc:2932] T a614380f92f64189ac5b5cdb42edf2e6 P cc5fffa9b8844c6f9c948344d8081833 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cc5fffa9b8844c6f9c948344d8081833" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45579 } } }
I20240503 01:59:10.646865 2253 catalog_manager.cc:5484] T a614380f92f64189ac5b5cdb42edf2e6 P cc5fffa9b8844c6f9c948344d8081833 reported cstate change: config changed from index 4 to 6, VOTER 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "cc5fffa9b8844c6f9c948344d8081833" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cc5fffa9b8844c6f9c948344d8081833" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45579 } } }
I20240503 01:59:10.650101 2510 tablet_service.cc:1514] Processing DeleteTablet for tablet a614380f92f64189ac5b5cdb42edf2e6 with delete_type TABLET_DATA_TOMBSTONED (TS 86ed23734e434f8784885ebf9f1423b2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44718
I20240503 01:59:10.650357 5353 tablet_replica.cc:330] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2: stopping tablet replica
I20240503 01:59:10.650488 5353 raft_consensus.cc:2229] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:10.650619 5353 raft_consensus.cc:2258] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:10.651055 5353 ts_tablet_manager.cc:1875] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:10.652875 5353 ts_tablet_manager.cc:1888] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 01:59:10.652959 5353 log.cc:1196] T a614380f92f64189ac5b5cdb42edf2e6 P 86ed23734e434f8784885ebf9f1423b2: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/a614380f92f64189ac5b5cdb42edf2e6
I20240503 01:59:10.653542 2242 catalog_manager.cc:4830] TS 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130:33335): tablet a614380f92f64189ac5b5cdb42edf2e6 (table rebalance_test_table_1 [id=2ecbe129f8fd43008007db41bf05979c]) successfully deleted
I20240503 01:59:10.716895 3776 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.721587 3514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.723325 2379 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:10.728071 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.735185 3643 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.735177 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.739095 2510 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:10.792745 3796 consensus_queue.cc:237] T 358379b5f00a41d69db831501091e702 P 03ea675b68a74c1192fa1a96f89a494b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "03ea675b68a74c1192fa1a96f89a494b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38459 } }
I20240503 01:59:10.793381 5276 raft_consensus.cc:2932] T 358379b5f00a41d69db831501091e702 P 03ea675b68a74c1192fa1a96f89a494b [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "03ea675b68a74c1192fa1a96f89a494b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38459 } } }
I20240503 01:59:10.796721 2253 catalog_manager.cc:5484] T 358379b5f00a41d69db831501091e702 P 03ea675b68a74c1192fa1a96f89a494b reported cstate change: config changed from index 4 to 6, VOTER 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "03ea675b68a74c1192fa1a96f89a494b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "03ea675b68a74c1192fa1a96f89a494b" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 38459 } } }
I20240503 01:59:10.799522 2379 tablet_service.cc:1514] Processing DeleteTablet for tablet 358379b5f00a41d69db831501091e702 with delete_type TABLET_DATA_TOMBSTONED (TS 80a019d9d1c34ccd9487207f90e68ab0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47770
I20240503 01:59:10.799809 5299 tablet_replica.cc:330] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0: stopping tablet replica
I20240503 01:59:10.799980 5299 raft_consensus.cc:2229] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:10.800102 5299 raft_consensus.cc:2258] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:10.800832 5299 ts_tablet_manager.cc:1875] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:10.803190 5299 ts_tablet_manager.cc:1888] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 01:59:10.803279 5299 log.cc:1196] T 358379b5f00a41d69db831501091e702 P 80a019d9d1c34ccd9487207f90e68ab0: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/358379b5f00a41d69db831501091e702
I20240503 01:59:10.804054 2242 catalog_manager.cc:4830] TS 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129:46305): tablet 358379b5f00a41d69db831501091e702 (table rebalance_test_table_2 [id=5e9078a735bc4854ab7fe5df318eca11]) successfully deleted
I20240503 01:59:10.866920 3776 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.871148 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.873422 3643 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.875896 3514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.879074 2379 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:10.879664 2510 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:10.902933 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:10.946197 3927 consensus_queue.cc:237] T 32f685df61f144628bbb19c89e36ee6e P f0ced467f93d42cdbb6c0a12a812b56a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f0ced467f93d42cdbb6c0a12a812b56a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33129 } }
I20240503 01:59:10.947151 5233 raft_consensus.cc:2932] T 32f685df61f144628bbb19c89e36ee6e P f0ced467f93d42cdbb6c0a12a812b56a [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f0ced467f93d42cdbb6c0a12a812b56a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33129 } } }
I20240503 01:59:10.950122 2257 catalog_manager.cc:5484] T 32f685df61f144628bbb19c89e36ee6e P f0ced467f93d42cdbb6c0a12a812b56a reported cstate change: config changed from index 4 to 6, VOTER 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "f0ced467f93d42cdbb6c0a12a812b56a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f0ced467f93d42cdbb6c0a12a812b56a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 33129 } } }
I20240503 01:59:10.953542 2379 tablet_service.cc:1514] Processing DeleteTablet for tablet 32f685df61f144628bbb19c89e36ee6e with delete_type TABLET_DATA_TOMBSTONED (TS 80a019d9d1c34ccd9487207f90e68ab0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47770
I20240503 01:59:10.953830 5299 tablet_replica.cc:330] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0: stopping tablet replica
I20240503 01:59:10.953989 5299 raft_consensus.cc:2229] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:10.954103 5299 raft_consensus.cc:2258] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:10.954759 5299 ts_tablet_manager.cc:1875] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:10.957108 5299 ts_tablet_manager.cc:1888] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 01:59:10.957249 5299 log.cc:1196] T 32f685df61f144628bbb19c89e36ee6e P 80a019d9d1c34ccd9487207f90e68ab0: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/32f685df61f144628bbb19c89e36ee6e
I20240503 01:59:10.958009 2242 catalog_manager.cc:4830] TS 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129:46305): tablet 32f685df61f144628bbb19c89e36ee6e (table rebalance_test_table_0 [id=d658689f115240b580f541db89933aa5]) successfully deleted
I20240503 01:59:11.029803 3776 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:11.030022 3643 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:11.030098 2379 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:11.037844 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:11.037982 3514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:11.042830 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:11.057087 2510 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:11.102102 3534 consensus_queue.cc:237] T 5c289e9ad385459aa89fa32637a42f36 P cc5fffa9b8844c6f9c948344d8081833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cc5fffa9b8844c6f9c948344d8081833" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45579 } }
I20240503 01:59:11.102811 5030 raft_consensus.cc:2932] T 5c289e9ad385459aa89fa32637a42f36 P cc5fffa9b8844c6f9c948344d8081833 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cc5fffa9b8844c6f9c948344d8081833" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45579 } } }
I20240503 01:59:11.105345 2255 catalog_manager.cc:5484] T 5c289e9ad385459aa89fa32637a42f36 P cc5fffa9b8844c6f9c948344d8081833 reported cstate change: config changed from index 4 to 6, VOTER 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "cc5fffa9b8844c6f9c948344d8081833" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cc5fffa9b8844c6f9c948344d8081833" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 45579 } } }
I20240503 01:59:11.108745 2510 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c289e9ad385459aa89fa32637a42f36 with delete_type TABLET_DATA_TOMBSTONED (TS 86ed23734e434f8784885ebf9f1423b2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44718
I20240503 01:59:11.109016 5353 tablet_replica.cc:330] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2: stopping tablet replica
I20240503 01:59:11.109519 5353 raft_consensus.cc:2229] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:11.109673 5353 raft_consensus.cc:2258] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:11.110481 5353 ts_tablet_manager.cc:1875] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:11.112872 5353 ts_tablet_manager.cc:1888] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 01:59:11.113014 5353 log.cc:1196] T 5c289e9ad385459aa89fa32637a42f36 P 86ed23734e434f8784885ebf9f1423b2: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/5c289e9ad385459aa89fa32637a42f36
I20240503 01:59:11.113595 2242 catalog_manager.cc:4830] TS 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130:33335): tablet 5c289e9ad385459aa89fa32637a42f36 (table rebalance_test_table_0 [id=d658689f115240b580f541db89933aa5]) successfully deleted
I20240503 01:59:11.172215 3643 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:11.173439 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:11.177292 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:11.179106 2379 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:11.181741 3776 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:11.189535 2510 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:11.195016 3514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:11.246958 3665 consensus_queue.cc:237] T ae344bce9aeb452b848accc96ddbb1ac P 65d5a1053fb5445486433fe66d82bdbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "65d5a1053fb5445486433fe66d82bdbe" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43855 } }
I20240503 01:59:11.247661 5610 raft_consensus.cc:2932] T ae344bce9aeb452b848accc96ddbb1ac P 65d5a1053fb5445486433fe66d82bdbe [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "65d5a1053fb5445486433fe66d82bdbe" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43855 } } }
I20240503 01:59:11.250257 2252 catalog_manager.cc:5484] T ae344bce9aeb452b848accc96ddbb1ac P 65d5a1053fb5445486433fe66d82bdbe reported cstate change: config changed from index 4 to 6, VOTER 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "65d5a1053fb5445486433fe66d82bdbe" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "65d5a1053fb5445486433fe66d82bdbe" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 43855 } } }
I20240503 01:59:11.253135 2379 tablet_service.cc:1514] Processing DeleteTablet for tablet ae344bce9aeb452b848accc96ddbb1ac with delete_type TABLET_DATA_TOMBSTONED (TS 80a019d9d1c34ccd9487207f90e68ab0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47770
I20240503 01:59:11.253391 5299 tablet_replica.cc:330] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0: stopping tablet replica
I20240503 01:59:11.253525 5299 raft_consensus.cc:2229] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:11.253639 5299 raft_consensus.cc:2258] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:11.254317 5299 ts_tablet_manager.cc:1875] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:11.256512 5299 ts_tablet_manager.cc:1888] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 01:59:11.256596 5299 log.cc:1196] T ae344bce9aeb452b848accc96ddbb1ac P 80a019d9d1c34ccd9487207f90e68ab0: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ae344bce9aeb452b848accc96ddbb1ac
I20240503 01:59:11.257176 2242 catalog_manager.cc:4830] TS 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129:46305): tablet ae344bce9aeb452b848accc96ddbb1ac (table rebalance_test_table_1 [id=2ecbe129f8fd43008007db41bf05979c]) successfully deleted
I20240503 01:59:11.575454 3776 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:11.575789 2379 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:11.576061 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:11.586019 3643 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:11.588811 2510 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:11.599393 3514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:11.600672 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:11.684510 3776 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:11.685657 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:11.687683 2510 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:11.687801 2379 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:11.692114 3514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:11.701560 3643 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:11.708894 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:11.950098 3643 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:11.952883 3776 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:11.953213 3514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:11.953215 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:11.953325 2510 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:11.955050 2379 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:11.956599 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:12.098203 3776 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:12.099328 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:12.103123 3514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:12.107828 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:12.123059 2510 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:12.124495 2379 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:12.125089 3643 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:12.235747 3643 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:12.241139 2379 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:12.243721 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:12.247884 3514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:12.256973 2510 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:12.263082 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:12.270413 3776 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
99706f940467469181bf2d4ab85588bb | 127.0.99.190:35781 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:35307 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
03ea675b68a74c1192fa1a96f89a494b | 127.0.99.133:38459 | HEALTHY | <none> | 2 | 0
5372f615183544f8bffc0fe13f4a152f | 127.0.99.135:45035 | HEALTHY | <none> | 2 | 0
65d5a1053fb5445486433fe66d82bdbe | 127.0.99.132:43855 | HEALTHY | <none> | 2 | 0
80a019d9d1c34ccd9487207f90e68ab0 | 127.0.99.129:46305 | HEALTHY | <none> | 3 | 0
86ed23734e434f8784885ebf9f1423b2 | 127.0.99.130:33335 | HEALTHY | <none> | 3 | 0
cc5fffa9b8844c6f9c948344d8081833 | 127.0.99.131:45579 | HEALTHY | <none> | 2 | 0
f0ced467f93d42cdbb6c0a12a812b56a | 127.0.99.134:33129 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:46305
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:33335
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:45579
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:43855
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:38459
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:33129
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:45035
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:46305
server_dump_info_path | /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:33335
server_dump_info_path | /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:45579
server_dump_info_path | /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:43855
server_dump_info_path | /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:38459
server_dump_info_path | /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:33129
server_dump_info_path | /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:45035
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:35307 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 01:59:12.301625 398 log_verifier.cc:126] Checking tablet 32f685df61f144628bbb19c89e36ee6e
I20240503 01:59:12.313177 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 32f685df61f144628bbb19c89e36ee6e
I20240503 01:59:12.313287 398 log_verifier.cc:126] Checking tablet 358379b5f00a41d69db831501091e702
I20240503 01:59:12.322803 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 358379b5f00a41d69db831501091e702
I20240503 01:59:12.322873 398 log_verifier.cc:126] Checking tablet 3e1fa0769a934b40a4a4ee536f96e5f4
I20240503 01:59:12.332638 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3e1fa0769a934b40a4a4ee536f96e5f4
I20240503 01:59:12.332711 398 log_verifier.cc:126] Checking tablet 3fc7ae29d38b4c539498cb946f1c9eaf
I20240503 01:59:12.341317 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 3fc7ae29d38b4c539498cb946f1c9eaf
I20240503 01:59:12.341392 398 log_verifier.cc:126] Checking tablet 49706355c8864e638e187eee7711cac3
I20240503 01:59:12.351512 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 49706355c8864e638e187eee7711cac3
I20240503 01:59:12.351585 398 log_verifier.cc:126] Checking tablet 5426821cad7e4b7b8689e5a6bfbad7ad
I20240503 01:59:12.362236 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 5426821cad7e4b7b8689e5a6bfbad7ad
I20240503 01:59:12.362308 398 log_verifier.cc:126] Checking tablet 5c289e9ad385459aa89fa32637a42f36
I20240503 01:59:12.372195 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 5c289e9ad385459aa89fa32637a42f36
I20240503 01:59:12.372269 398 log_verifier.cc:126] Checking tablet 5f821e4f2b37483abda9bd5a89cd0b19
I20240503 01:59:12.381906 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 5f821e4f2b37483abda9bd5a89cd0b19
I20240503 01:59:12.381968 398 log_verifier.cc:126] Checking tablet 7710730e9d5b440d98bd522c2f808fa3
I20240503 01:59:12.391840 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7710730e9d5b440d98bd522c2f808fa3
I20240503 01:59:12.391918 398 log_verifier.cc:126] Checking tablet 7d729ec44ced4c01a4b84de1fcac3a6f
I20240503 01:59:12.402062 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7d729ec44ced4c01a4b84de1fcac3a6f
I20240503 01:59:12.402129 398 log_verifier.cc:126] Checking tablet 833d9ce859f84b499e667d86965d9fff
I20240503 01:59:12.412009 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 833d9ce859f84b499e667d86965d9fff
I20240503 01:59:12.412094 398 log_verifier.cc:126] Checking tablet 986d142f7be641ed9a9c338c41ba430a
I20240503 01:59:12.422286 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 986d142f7be641ed9a9c338c41ba430a
I20240503 01:59:12.422366 398 log_verifier.cc:126] Checking tablet a614380f92f64189ac5b5cdb42edf2e6
I20240503 01:59:12.432483 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet a614380f92f64189ac5b5cdb42edf2e6
I20240503 01:59:12.432560 398 log_verifier.cc:126] Checking tablet ae344bce9aeb452b848accc96ddbb1ac
I20240503 01:59:12.442695 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet ae344bce9aeb452b848accc96ddbb1ac
I20240503 01:59:12.442762 398 log_verifier.cc:126] Checking tablet c0bce60681a34a96a83284c762d6feee
I20240503 01:59:12.452682 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet c0bce60681a34a96a83284c762d6feee
I20240503 01:59:12.452754 398 log_verifier.cc:126] Checking tablet f586b7fd77824920b4300f38a29ba87b
I20240503 01:59:12.462468 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet f586b7fd77824920b4300f38a29ba87b
I20240503 01:59:12.464673 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35781
--builtin_ntp_servers=127.0.99.148:35307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 01:59:12.578591 5947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:12.578842 5947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:12.578888 5947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:12.582438 5947 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:59:12.582572 5947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:12.582686 5947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 01:59:12.586671 5947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:35781
--never_fsync=true
--heap_profile_path=/tmp/kudu.5947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:12.587658 5947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:12.588793 5947 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:12.598316 5955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:12.599025 5952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:12.599776 5953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:12.601081 5947 server_base.cc:1034] running on GCE node
I20240503 01:59:12.601626 5947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:12.602290 5947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:12.603551 5947 hybrid_clock.cc:648] HybridClock initialized: now 1714701552603518 us; error 54 us; skew 500 ppm
I20240503 01:59:12.605767 5947 webserver.cc:460] Webserver started at http://127.0.99.136:37489/ using document root <none> and password file <none>
I20240503 01:59:12.606353 5947 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:12.606420 5947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:12.606648 5947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:12.608388 5947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "08fcdb7701204e9795ebccb5b4b8e669"
format_stamp: "Formatted at 2024-05-03 01:59:12 on dist-test-slave-w9tw"
I20240503 01:59:12.608896 5947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "08fcdb7701204e9795ebccb5b4b8e669"
format_stamp: "Formatted at 2024-05-03 01:59:12 on dist-test-slave-w9tw"
I20240503 01:59:12.612601 5947 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:59:12.615185 5961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:12.616582 5947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:12.616806 5947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "08fcdb7701204e9795ebccb5b4b8e669"
format_stamp: "Formatted at 2024-05-03 01:59:12 on dist-test-slave-w9tw"
I20240503 01:59:12.616920 5947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:12.641861 5947 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:12.642593 5947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:12.642756 5947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:12.643314 5947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:12.644316 5947 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:12.644367 5947 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:12.644424 5947 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:12.644450 5947 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:12.654780 5947 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:39515
I20240503 01:59:12.654800 6074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:39515 every 8 connection(s)
I20240503 01:59:12.656172 5947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 01:59:12.661146 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 5947
I20240503 01:59:12.661254 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 01:59:12.665707 6075 heartbeater.cc:347] Connected to a master server at 127.0.99.190:35781
I20240503 01:59:12.666045 6075 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:12.666851 6075 heartbeater.cc:511] Master 127.0.99.190:35781 requested a full tablet report, sending...
I20240503 01:59:12.668242 2252 ts_manager.cc:194] Registered new tserver with Master: 08fcdb7701204e9795ebccb5b4b8e669 (127.0.99.136:39515)
I20240503 01:59:12.668963 2252 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:37007
I20240503 01:59:12.862754 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:12.863128 6009 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:12.863538 3776 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:12.866590 2379 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:12.872663 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:12.874717 2510 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:12.883136 3643 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:12.912716 3514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:13.039322 2510 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:13.042459 3776 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:13.043278 2379 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:13.045356 3643 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:13.045784 6009 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:13.059685 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:13.061571 3514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:13.080901 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:13.125021 2530 consensus_queue.cc:237] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "08fcdb7701204e9795ebccb5b4b8e669" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39515 } }
W20240503 01:59:13.130254 2466 consensus_peers.cc:485] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2 -> Peer 08fcdb7701204e9795ebccb5b4b8e669 (127.0.99.136:39515): Couldn't send request to peer 08fcdb7701204e9795ebccb5b4b8e669. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5426821cad7e4b7b8689e5a6bfbad7ad. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:13.586661 6180 ts_tablet_manager.cc:902] T 5426821cad7e4b7b8689e5a6bfbad7ad P 08fcdb7701204e9795ebccb5b4b8e669: Initiating tablet copy from peer 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130:33335)
I20240503 01:59:13.587359 6180 tablet_copy_client.cc:308] T 5426821cad7e4b7b8689e5a6bfbad7ad P 08fcdb7701204e9795ebccb5b4b8e669: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:33335
I20240503 01:59:13.591264 2550 tablet_copy_service.cc:140] P 86ed23734e434f8784885ebf9f1423b2: Received BeginTabletCopySession request for tablet 5426821cad7e4b7b8689e5a6bfbad7ad from peer 08fcdb7701204e9795ebccb5b4b8e669 ({username='slave'} at 127.0.99.136:50443)
I20240503 01:59:13.591383 2550 tablet_copy_service.cc:161] P 86ed23734e434f8784885ebf9f1423b2: Beginning new tablet copy session on tablet 5426821cad7e4b7b8689e5a6bfbad7ad from peer 08fcdb7701204e9795ebccb5b4b8e669 at {username='slave'} at 127.0.99.136:50443: session id = 08fcdb7701204e9795ebccb5b4b8e669-5426821cad7e4b7b8689e5a6bfbad7ad
I20240503 01:59:13.592144 2550 tablet_copy_source_session.cc:205] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:13.595198 6180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5426821cad7e4b7b8689e5a6bfbad7ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:13.600034 6180 tablet_copy_client.cc:783] T 5426821cad7e4b7b8689e5a6bfbad7ad P 08fcdb7701204e9795ebccb5b4b8e669: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:13.600227 6180 tablet_copy_client.cc:647] T 5426821cad7e4b7b8689e5a6bfbad7ad P 08fcdb7701204e9795ebccb5b4b8e669: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:13.601780 6180 tablet_copy_client.cc:523] T 5426821cad7e4b7b8689e5a6bfbad7ad P 08fcdb7701204e9795ebccb5b4b8e669: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:13.641499 6180 tablet_bootstrap.cc:492] T 5426821cad7e4b7b8689e5a6bfbad7ad P 08fcdb7701204e9795ebccb5b4b8e669: Bootstrap starting.
I20240503 01:59:13.647303 6180 log.cc:824] T 5426821cad7e4b7b8689e5a6bfbad7ad P 08fcdb7701204e9795ebccb5b4b8e669: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:13.668864 6180 tablet_bootstrap.cc:492] T 5426821cad7e4b7b8689e5a6bfbad7ad P 08fcdb7701204e9795ebccb5b4b8e669: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 01:59:13.669289 6180 tablet_bootstrap.cc:492] T 5426821cad7e4b7b8689e5a6bfbad7ad P 08fcdb7701204e9795ebccb5b4b8e669: Bootstrap complete.
I20240503 01:59:13.669610 6180 ts_tablet_manager.cc:1367] T 5426821cad7e4b7b8689e5a6bfbad7ad P 08fcdb7701204e9795ebccb5b4b8e669: Time spent bootstrapping tablet: real 0.028s user 0.022s sys 0.004s
I20240503 01:59:13.671315 6075 heartbeater.cc:503] Master 127.0.99.190:35781 was elected leader, sending a full tablet report...
I20240503 01:59:13.672045 6180 raft_consensus.cc:348] T 5426821cad7e4b7b8689e5a6bfbad7ad P 08fcdb7701204e9795ebccb5b4b8e669 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } }
I20240503 01:59:13.672652 6180 raft_consensus.cc:729] T 5426821cad7e4b7b8689e5a6bfbad7ad P 08fcdb7701204e9795ebccb5b4b8e669 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08fcdb7701204e9795ebccb5b4b8e669, State: Initialized, Role: FOLLOWER
I20240503 01:59:13.672951 6180 consensus_queue.cc:260] T 5426821cad7e4b7b8689e5a6bfbad7ad P 08fcdb7701204e9795ebccb5b4b8e669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "08fcdb7701204e9795ebccb5b4b8e669" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39515 } }
I20240503 01:59:13.673859 6180 ts_tablet_manager.cc:1398] T 5426821cad7e4b7b8689e5a6bfbad7ad P 08fcdb7701204e9795ebccb5b4b8e669: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20240503 01:59:13.674577 2550 tablet_copy_service.cc:342] P 86ed23734e434f8784885ebf9f1423b2: Request end of tablet copy session 08fcdb7701204e9795ebccb5b4b8e669-5426821cad7e4b7b8689e5a6bfbad7ad received from {username='slave'} at 127.0.99.136:50443
I20240503 01:59:13.674662 2550 tablet_copy_service.cc:433] P 86ed23734e434f8784885ebf9f1423b2: ending tablet copy session 08fcdb7701204e9795ebccb5b4b8e669-5426821cad7e4b7b8689e5a6bfbad7ad on tablet 5426821cad7e4b7b8689e5a6bfbad7ad with peer 08fcdb7701204e9795ebccb5b4b8e669
I20240503 01:59:14.113569 6029 raft_consensus.cc:1203] T 5426821cad7e4b7b8689e5a6bfbad7ad P 08fcdb7701204e9795ebccb5b4b8e669 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:14.114917 6189 raft_consensus.cc:2932] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 08fcdb7701204e9795ebccb5b4b8e669 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "08fcdb7701204e9795ebccb5b4b8e669" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39515 } } }
I20240503 01:59:14.115429 6029 raft_consensus.cc:2932] T 5426821cad7e4b7b8689e5a6bfbad7ad P 08fcdb7701204e9795ebccb5b4b8e669 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 08fcdb7701204e9795ebccb5b4b8e669 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "08fcdb7701204e9795ebccb5b4b8e669" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39515 } } }
I20240503 01:59:14.170627 6189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:42968 (request call id 0) took 1045 ms. Trace:
I20240503 01:59:14.170806 6189 rpcz_store.cc:276] 0503 01:59:13.124636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 01:59:13.124694 (+ 58us) service_pool.cc:225] Handling call
0503 01:59:13.125491 (+ 797us) log.cc:842] Serialized 257 byte log entry
0503 01:59:14.170230 (+1044739us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":48,"raft.queue_time_us":108,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":119,"thread_start_us":148,"threads_started":2,"wal-append.queue_time_us":216,"wal-append.run_cpu_time_us":434,"wal-append.run_wall_time_us":1000625}
I20240503 01:59:14.172641 2256 catalog_manager.cc:5484] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2 reported cstate change: config changed from index -1 to 4, VOTER 08fcdb7701204e9795ebccb5b4b8e669 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "86ed23734e434f8784885ebf9f1423b2" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "08fcdb7701204e9795ebccb5b4b8e669" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39515 } } }
I20240503 01:59:14.194003 2399 consensus_queue.cc:237] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "08fcdb7701204e9795ebccb5b4b8e669" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39515 } }
W20240503 01:59:14.198938 2335 consensus_peers.cc:485] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0 -> Peer 08fcdb7701204e9795ebccb5b4b8e669 (127.0.99.136:39515): Couldn't send request to peer 08fcdb7701204e9795ebccb5b4b8e669. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7710730e9d5b440d98bd522c2f808fa3. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:14.661423 6217 ts_tablet_manager.cc:902] T 7710730e9d5b440d98bd522c2f808fa3 P 08fcdb7701204e9795ebccb5b4b8e669: Initiating tablet copy from peer 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129:46305)
I20240503 01:59:14.661882 6217 tablet_copy_client.cc:308] T 7710730e9d5b440d98bd522c2f808fa3 P 08fcdb7701204e9795ebccb5b4b8e669: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:46305
I20240503 01:59:14.665530 2419 tablet_copy_service.cc:140] P 80a019d9d1c34ccd9487207f90e68ab0: Received BeginTabletCopySession request for tablet 7710730e9d5b440d98bd522c2f808fa3 from peer 08fcdb7701204e9795ebccb5b4b8e669 ({username='slave'} at 127.0.99.136:59437)
I20240503 01:59:14.665642 2419 tablet_copy_service.cc:161] P 80a019d9d1c34ccd9487207f90e68ab0: Beginning new tablet copy session on tablet 7710730e9d5b440d98bd522c2f808fa3 from peer 08fcdb7701204e9795ebccb5b4b8e669 at {username='slave'} at 127.0.99.136:59437: session id = 08fcdb7701204e9795ebccb5b4b8e669-7710730e9d5b440d98bd522c2f808fa3
I20240503 01:59:14.666359 2419 tablet_copy_source_session.cc:205] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:14.667276 6217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7710730e9d5b440d98bd522c2f808fa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:14.669627 6217 tablet_copy_client.cc:783] T 7710730e9d5b440d98bd522c2f808fa3 P 08fcdb7701204e9795ebccb5b4b8e669: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:14.669852 6217 tablet_copy_client.cc:647] T 7710730e9d5b440d98bd522c2f808fa3 P 08fcdb7701204e9795ebccb5b4b8e669: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:14.671574 6217 tablet_copy_client.cc:523] T 7710730e9d5b440d98bd522c2f808fa3 P 08fcdb7701204e9795ebccb5b4b8e669: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:14.672960 6217 tablet_bootstrap.cc:492] T 7710730e9d5b440d98bd522c2f808fa3 P 08fcdb7701204e9795ebccb5b4b8e669: Bootstrap starting.
I20240503 01:59:14.692152 6217 tablet_bootstrap.cc:492] T 7710730e9d5b440d98bd522c2f808fa3 P 08fcdb7701204e9795ebccb5b4b8e669: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 01:59:14.692700 6217 tablet_bootstrap.cc:492] T 7710730e9d5b440d98bd522c2f808fa3 P 08fcdb7701204e9795ebccb5b4b8e669: Bootstrap complete.
I20240503 01:59:14.692975 6217 ts_tablet_manager.cc:1367] T 7710730e9d5b440d98bd522c2f808fa3 P 08fcdb7701204e9795ebccb5b4b8e669: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.005s
I20240503 01:59:14.693588 6217 raft_consensus.cc:348] T 7710730e9d5b440d98bd522c2f808fa3 P 08fcdb7701204e9795ebccb5b4b8e669 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } }
I20240503 01:59:14.693836 6217 raft_consensus.cc:729] T 7710730e9d5b440d98bd522c2f808fa3 P 08fcdb7701204e9795ebccb5b4b8e669 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08fcdb7701204e9795ebccb5b4b8e669, State: Initialized, Role: FOLLOWER
I20240503 01:59:14.693965 6217 consensus_queue.cc:260] T 7710730e9d5b440d98bd522c2f808fa3 P 08fcdb7701204e9795ebccb5b4b8e669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "08fcdb7701204e9795ebccb5b4b8e669" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39515 } }
I20240503 01:59:14.694339 6217 ts_tablet_manager.cc:1398] T 7710730e9d5b440d98bd522c2f808fa3 P 08fcdb7701204e9795ebccb5b4b8e669: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 01:59:14.694999 2419 tablet_copy_service.cc:342] P 80a019d9d1c34ccd9487207f90e68ab0: Request end of tablet copy session 08fcdb7701204e9795ebccb5b4b8e669-7710730e9d5b440d98bd522c2f808fa3 received from {username='slave'} at 127.0.99.136:59437
I20240503 01:59:14.695083 2419 tablet_copy_service.cc:433] P 80a019d9d1c34ccd9487207f90e68ab0: ending tablet copy session 08fcdb7701204e9795ebccb5b4b8e669-7710730e9d5b440d98bd522c2f808fa3 on tablet 7710730e9d5b440d98bd522c2f808fa3 with peer 08fcdb7701204e9795ebccb5b4b8e669
I20240503 01:59:15.139941 6029 raft_consensus.cc:1203] T 7710730e9d5b440d98bd522c2f808fa3 P 08fcdb7701204e9795ebccb5b4b8e669 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:15.140975 6226 raft_consensus.cc:2932] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 08fcdb7701204e9795ebccb5b4b8e669 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "08fcdb7701204e9795ebccb5b4b8e669" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39515 } } }
I20240503 01:59:15.141451 6029 raft_consensus.cc:2932] T 7710730e9d5b440d98bd522c2f808fa3 P 08fcdb7701204e9795ebccb5b4b8e669 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 08fcdb7701204e9795ebccb5b4b8e669 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "08fcdb7701204e9795ebccb5b4b8e669" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39515 } } }
I20240503 01:59:15.144066 2252 catalog_manager.cc:5484] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0 reported cstate change: config changed from index -1 to 4, VOTER 08fcdb7701204e9795ebccb5b4b8e669 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "80a019d9d1c34ccd9487207f90e68ab0" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "08fcdb7701204e9795ebccb5b4b8e669" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39515 } } }
I20240503 01:59:15.212968 6009 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:15.214167 3776 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.215451 2379 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:15.219228 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.219952 3643 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.225082 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.227116 2510 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:15.255518 3514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.281105 2399 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7710730e9d5b440d98bd522c2f808fa3"
dest_uuid: "80a019d9d1c34ccd9487207f90e68ab0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44458
I20240503 01:59:15.281318 2399 raft_consensus.cc:595] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:15.344223 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.344590 3776 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.346400 2379 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:15.353255 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.356284 3514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.357553 3643 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.359277 6009 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:15.376492 2510 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:15.468277 6009 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:15.469077 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.473031 2379 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:15.473853 3776 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.487779 3643 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.495303 2510 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:15.507437 3514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.507767 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.533562 2530 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5426821cad7e4b7b8689e5a6bfbad7ad"
dest_uuid: "86ed23734e434f8784885ebf9f1423b2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43014
I20240503 01:59:15.533716 2530 raft_consensus.cc:595] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:15.593554 3776 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.593854 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.596848 2379 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:15.599495 3514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.599962 3643 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.602713 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.607628 6009 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:15.615249 2510 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:15.758395 6392 raft_consensus.cc:979] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0: : Instructing follower 08fcdb7701204e9795ebccb5b4b8e669 to start an election
I20240503 01:59:15.758493 6392 raft_consensus.cc:1067] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Signalling peer 08fcdb7701204e9795ebccb5b4b8e669 to start an election
I20240503 01:59:15.758975 6029 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7710730e9d5b440d98bd522c2f808fa3"
dest_uuid: "08fcdb7701204e9795ebccb5b4b8e669"
from {username='slave'} at 127.0.99.129:39137
I20240503 01:59:15.759153 6029 raft_consensus.cc:482] T 7710730e9d5b440d98bd522c2f808fa3 P 08fcdb7701204e9795ebccb5b4b8e669 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:15.759253 6029 raft_consensus.cc:3037] T 7710730e9d5b440d98bd522c2f808fa3 P 08fcdb7701204e9795ebccb5b4b8e669 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:15.760074 6029 raft_consensus.cc:504] T 7710730e9d5b440d98bd522c2f808fa3 P 08fcdb7701204e9795ebccb5b4b8e669 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "08fcdb7701204e9795ebccb5b4b8e669" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39515 } }
I20240503 01:59:15.760936 6029 leader_election.cc:288] T 7710730e9d5b440d98bd522c2f808fa3 P 08fcdb7701204e9795ebccb5b4b8e669 [CANDIDATE]: Term 2 election: Requested vote from peers 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129:46305)
I20240503 01:59:15.761538 2399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7710730e9d5b440d98bd522c2f808fa3" candidate_uuid: "08fcdb7701204e9795ebccb5b4b8e669" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "80a019d9d1c34ccd9487207f90e68ab0"
I20240503 01:59:15.761667 2399 raft_consensus.cc:3032] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:15.761698 2399 raft_consensus.cc:729] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 80a019d9d1c34ccd9487207f90e68ab0, State: Running, Role: LEADER
I20240503 01:59:15.761871 2399 consensus_queue.cc:260] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "08fcdb7701204e9795ebccb5b4b8e669" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39515 } }
I20240503 01:59:15.762018 2399 raft_consensus.cc:3037] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:15.762954 2399 raft_consensus.cc:2454] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08fcdb7701204e9795ebccb5b4b8e669 in term 2.
I20240503 01:59:15.763381 5963 leader_election.cc:302] T 7710730e9d5b440d98bd522c2f808fa3 P 08fcdb7701204e9795ebccb5b4b8e669 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 08fcdb7701204e9795ebccb5b4b8e669, 80a019d9d1c34ccd9487207f90e68ab0; no voters:
I20240503 01:59:15.763749 6394 raft_consensus.cc:2781] T 7710730e9d5b440d98bd522c2f808fa3 P 08fcdb7701204e9795ebccb5b4b8e669 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:15.763926 6394 raft_consensus.cc:686] T 7710730e9d5b440d98bd522c2f808fa3 P 08fcdb7701204e9795ebccb5b4b8e669 [term 2 LEADER]: Becoming Leader. State: Replica: 08fcdb7701204e9795ebccb5b4b8e669, State: Running, Role: LEADER
I20240503 01:59:15.764250 6394 consensus_queue.cc:237] T 7710730e9d5b440d98bd522c2f808fa3 P 08fcdb7701204e9795ebccb5b4b8e669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "08fcdb7701204e9795ebccb5b4b8e669" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39515 } }
I20240503 01:59:15.766666 2256 catalog_manager.cc:5484] T 7710730e9d5b440d98bd522c2f808fa3 P 08fcdb7701204e9795ebccb5b4b8e669 reported cstate change: term changed from 1 to 2, leader changed from 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129) to 08fcdb7701204e9795ebccb5b4b8e669 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "08fcdb7701204e9795ebccb5b4b8e669" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 } } peers { permanent_uuid: "08fcdb7701204e9795ebccb5b4b8e669" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39515 } } }
I20240503 01:59:15.869035 6315 raft_consensus.cc:979] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2: : Instructing follower 08fcdb7701204e9795ebccb5b4b8e669 to start an election
I20240503 01:59:15.869167 6315 raft_consensus.cc:1067] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Signalling peer 08fcdb7701204e9795ebccb5b4b8e669 to start an election
I20240503 01:59:15.869627 6029 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5426821cad7e4b7b8689e5a6bfbad7ad"
dest_uuid: "08fcdb7701204e9795ebccb5b4b8e669"
from {username='slave'} at 127.0.99.130:52861
I20240503 01:59:15.869781 6029 raft_consensus.cc:482] T 5426821cad7e4b7b8689e5a6bfbad7ad P 08fcdb7701204e9795ebccb5b4b8e669 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:15.869853 6029 raft_consensus.cc:3037] T 5426821cad7e4b7b8689e5a6bfbad7ad P 08fcdb7701204e9795ebccb5b4b8e669 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:15.870921 6029 raft_consensus.cc:504] T 5426821cad7e4b7b8689e5a6bfbad7ad P 08fcdb7701204e9795ebccb5b4b8e669 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "08fcdb7701204e9795ebccb5b4b8e669" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39515 } }
I20240503 01:59:15.871335 6029 leader_election.cc:288] T 5426821cad7e4b7b8689e5a6bfbad7ad P 08fcdb7701204e9795ebccb5b4b8e669 [CANDIDATE]: Term 2 election: Requested vote from peers 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130:33335)
I20240503 01:59:15.871778 2530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5426821cad7e4b7b8689e5a6bfbad7ad" candidate_uuid: "08fcdb7701204e9795ebccb5b4b8e669" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "86ed23734e434f8784885ebf9f1423b2"
I20240503 01:59:15.871910 2530 raft_consensus.cc:3032] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:15.871987 2530 raft_consensus.cc:729] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 86ed23734e434f8784885ebf9f1423b2, State: Running, Role: LEADER
I20240503 01:59:15.872164 2530 consensus_queue.cc:260] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "08fcdb7701204e9795ebccb5b4b8e669" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39515 } }
I20240503 01:59:15.872311 2530 raft_consensus.cc:3037] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:15.873149 2530 raft_consensus.cc:2454] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08fcdb7701204e9795ebccb5b4b8e669 in term 2.
I20240503 01:59:15.873450 5963 leader_election.cc:302] T 5426821cad7e4b7b8689e5a6bfbad7ad P 08fcdb7701204e9795ebccb5b4b8e669 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 08fcdb7701204e9795ebccb5b4b8e669, 86ed23734e434f8784885ebf9f1423b2; no voters:
I20240503 01:59:15.873668 6394 raft_consensus.cc:2781] T 5426821cad7e4b7b8689e5a6bfbad7ad P 08fcdb7701204e9795ebccb5b4b8e669 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:15.873755 6394 raft_consensus.cc:686] T 5426821cad7e4b7b8689e5a6bfbad7ad P 08fcdb7701204e9795ebccb5b4b8e669 [term 2 LEADER]: Becoming Leader. State: Replica: 08fcdb7701204e9795ebccb5b4b8e669, State: Running, Role: LEADER
I20240503 01:59:15.873901 6394 consensus_queue.cc:237] T 5426821cad7e4b7b8689e5a6bfbad7ad P 08fcdb7701204e9795ebccb5b4b8e669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "08fcdb7701204e9795ebccb5b4b8e669" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39515 } }
I20240503 01:59:15.875495 2256 catalog_manager.cc:5484] T 5426821cad7e4b7b8689e5a6bfbad7ad P 08fcdb7701204e9795ebccb5b4b8e669 reported cstate change: term changed from 1 to 2, leader changed from 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130) to 08fcdb7701204e9795ebccb5b4b8e669 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "08fcdb7701204e9795ebccb5b4b8e669" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 } } peers { permanent_uuid: "08fcdb7701204e9795ebccb5b4b8e669" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39515 } } }
I20240503 01:59:15.918407 2510 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.919090 3776 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.921690 6009 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.927807 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.936138 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.941869 3643 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.942323 3514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:15.955538 2379 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.039932 3776 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.042672 6009 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.044210 2379 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.046274 2510 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.047922 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.054713 3514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.057598 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.064440 3643 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.226300 2399 raft_consensus.cc:1261] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 2 FOLLOWER]: Refusing update from remote peer 08fcdb7701204e9795ebccb5b4b8e669: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 01:59:16.226898 6394 consensus_queue.cc:1035] T 7710730e9d5b440d98bd522c2f808fa3 P 08fcdb7701204e9795ebccb5b4b8e669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80a019d9d1c34ccd9487207f90e68ab0" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 46305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:16.264981 2530 raft_consensus.cc:1261] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2 [term 2 FOLLOWER]: Refusing update from remote peer 08fcdb7701204e9795ebccb5b4b8e669: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 01:59:16.265497 6395 consensus_queue.cc:1035] T 5426821cad7e4b7b8689e5a6bfbad7ad P 08fcdb7701204e9795ebccb5b4b8e669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86ed23734e434f8784885ebf9f1423b2" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 33335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:16.368052 6009 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.368674 2510 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.370184 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.376574 2379 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.384038 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.384822 3776 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.395547 3643 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.414319 3514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.452127 6029 consensus_queue.cc:237] T 7710730e9d5b440d98bd522c2f808fa3 P 08fcdb7701204e9795ebccb5b4b8e669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "08fcdb7701204e9795ebccb5b4b8e669" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39515 } }
I20240503 01:59:16.452733 6394 raft_consensus.cc:2932] T 7710730e9d5b440d98bd522c2f808fa3 P 08fcdb7701204e9795ebccb5b4b8e669 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "08fcdb7701204e9795ebccb5b4b8e669" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39515 } } }
I20240503 01:59:16.455294 2252 catalog_manager.cc:5484] T 7710730e9d5b440d98bd522c2f808fa3 P 08fcdb7701204e9795ebccb5b4b8e669 reported cstate change: config changed from index 4 to 6, VOTER 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "08fcdb7701204e9795ebccb5b4b8e669" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "08fcdb7701204e9795ebccb5b4b8e669" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39515 } } }
I20240503 01:59:16.458670 2379 tablet_service.cc:1514] Processing DeleteTablet for tablet 7710730e9d5b440d98bd522c2f808fa3 with delete_type TABLET_DATA_TOMBSTONED (TS 80a019d9d1c34ccd9487207f90e68ab0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47770
I20240503 01:59:16.459095 6543 tablet_replica.cc:330] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0: stopping tablet replica
I20240503 01:59:16.459244 6543 raft_consensus.cc:2229] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:16.459379 6543 raft_consensus.cc:2258] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:16.460726 6543 ts_tablet_manager.cc:1875] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:16.462761 6543 ts_tablet_manager.cc:1888] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 01:59:16.462841 6543 log.cc:1196] T 7710730e9d5b440d98bd522c2f808fa3 P 80a019d9d1c34ccd9487207f90e68ab0: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/7710730e9d5b440d98bd522c2f808fa3
I20240503 01:59:16.463413 2242 catalog_manager.cc:4830] TS 80a019d9d1c34ccd9487207f90e68ab0 (127.0.99.129:46305): tablet 7710730e9d5b440d98bd522c2f808fa3 (table rebalance_test_table_2 [id=5e9078a735bc4854ab7fe5df318eca11]) successfully deleted
I20240503 01:59:16.524335 6009 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.526924 3776 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.531430 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.534235 3643 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.534585 3514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.542059 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.546875 2510 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.560804 2379 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.608250 6029 consensus_queue.cc:237] T 5426821cad7e4b7b8689e5a6bfbad7ad P 08fcdb7701204e9795ebccb5b4b8e669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "08fcdb7701204e9795ebccb5b4b8e669" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39515 } }
I20240503 01:59:16.608929 6394 raft_consensus.cc:2932] T 5426821cad7e4b7b8689e5a6bfbad7ad P 08fcdb7701204e9795ebccb5b4b8e669 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "08fcdb7701204e9795ebccb5b4b8e669" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39515 } } }
I20240503 01:59:16.611837 2248 catalog_manager.cc:5484] T 5426821cad7e4b7b8689e5a6bfbad7ad P 08fcdb7701204e9795ebccb5b4b8e669 reported cstate change: config changed from index 4 to 6, VOTER 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130) evicted. New cstate: current_term: 2 leader_uuid: "08fcdb7701204e9795ebccb5b4b8e669" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "08fcdb7701204e9795ebccb5b4b8e669" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 39515 } } }
I20240503 01:59:16.614735 2510 tablet_service.cc:1514] Processing DeleteTablet for tablet 5426821cad7e4b7b8689e5a6bfbad7ad with delete_type TABLET_DATA_TOMBSTONED (TS 86ed23734e434f8784885ebf9f1423b2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:44718
I20240503 01:59:16.615146 6592 tablet_replica.cc:330] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2: stopping tablet replica
I20240503 01:59:16.615339 6592 raft_consensus.cc:2229] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:16.615458 6592 raft_consensus.cc:2258] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:16.616878 6592 ts_tablet_manager.cc:1875] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:16.618732 6592 ts_tablet_manager.cc:1888] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 01:59:16.618819 6592 log.cc:1196] T 5426821cad7e4b7b8689e5a6bfbad7ad P 86ed23734e434f8784885ebf9f1423b2: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/5426821cad7e4b7b8689e5a6bfbad7ad
I20240503 01:59:16.619457 2242 catalog_manager.cc:4830] TS 86ed23734e434f8784885ebf9f1423b2 (127.0.99.130:33335): tablet 5426821cad7e4b7b8689e5a6bfbad7ad (table rebalance_test_table_3 [id=cc6ae6794e364638a2b3a21c6e7329ea]) successfully deleted
I20240503 01:59:16.887621 2379 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.893376 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.897573 3643 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.898321 3776 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.898788 6009 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.910452 3514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.914342 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:16.926268 2510 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:17.018821 6009 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:17.023710 3643 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:17.025043 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:17.028537 3776 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:17.031515 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:17.038621 2379 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:17.044518 3514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:17.047582 2510 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:17.159449 4039 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:17.161020 6009 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:17.167590 3776 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:17.169564 2510 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:17.175685 3643 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:17.182651 3514 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:17.183651 3907 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:17.202131 2379 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
99706f940467469181bf2d4ab85588bb | 127.0.99.190:35781 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:35307 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
03ea675b68a74c1192fa1a96f89a494b | 127.0.99.133:38459 | HEALTHY | <none> | 2 | 0
08fcdb7701204e9795ebccb5b4b8e669 | 127.0.99.136:39515 | HEALTHY | <none> | 2 | 0
5372f615183544f8bffc0fe13f4a152f | 127.0.99.135:45035 | HEALTHY | <none> | 2 | 0
65d5a1053fb5445486433fe66d82bdbe | 127.0.99.132:43855 | HEALTHY | <none> | 2 | 0
80a019d9d1c34ccd9487207f90e68ab0 | 127.0.99.129:46305 | HEALTHY | <none> | 2 | 0
86ed23734e434f8784885ebf9f1423b2 | 127.0.99.130:33335 | HEALTHY | <none> | 2 | 0
cc5fffa9b8844c6f9c948344d8081833 | 127.0.99.131:45579 | HEALTHY | <none> | 2 | 0
f0ced467f93d42cdbb6c0a12a812b56a | 127.0.99.134:33129 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.0.99.129 | experimental | 127.0.99.129:46305
local_ip_for_outbound_sockets | 127.0.99.130 | experimental | 127.0.99.130:33335
local_ip_for_outbound_sockets | 127.0.99.131 | experimental | 127.0.99.131:45579
local_ip_for_outbound_sockets | 127.0.99.132 | experimental | 127.0.99.132:43855
local_ip_for_outbound_sockets | 127.0.99.133 | experimental | 127.0.99.133:38459
local_ip_for_outbound_sockets | 127.0.99.134 | experimental | 127.0.99.134:33129
local_ip_for_outbound_sockets | 127.0.99.135 | experimental | 127.0.99.135:45035
local_ip_for_outbound_sockets | 127.0.99.136 | experimental | 127.0.99.136:39515
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.99.129:46305
server_dump_info_path | /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.99.130:33335
server_dump_info_path | /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.99.131:45579
server_dump_info_path | /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.99.132:43855
server_dump_info_path | /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.99.133:38459
server_dump_info_path | /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.99.134:33129
server_dump_info_path | /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.99.135:45035
server_dump_info_path | /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.0.99.136:39515
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.99.148:35307 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 01:59:17.225201 398 log_verifier.cc:126] Checking tablet 32f685df61f144628bbb19c89e36ee6e
I20240503 01:59:17.232050 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 32f685df61f144628bbb19c89e36ee6e
I20240503 01:59:17.232124 398 log_verifier.cc:126] Checking tablet 358379b5f00a41d69db831501091e702
I20240503 01:59:17.238198 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 358379b5f00a41d69db831501091e702
I20240503 01:59:17.238268 398 log_verifier.cc:126] Checking tablet 3e1fa0769a934b40a4a4ee536f96e5f4
I20240503 01:59:17.244011 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3e1fa0769a934b40a4a4ee536f96e5f4
I20240503 01:59:17.244150 398 log_verifier.cc:126] Checking tablet 3fc7ae29d38b4c539498cb946f1c9eaf
I20240503 01:59:17.249040 398 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 3fc7ae29d38b4c539498cb946f1c9eaf
I20240503 01:59:17.249102 398 log_verifier.cc:126] Checking tablet 49706355c8864e638e187eee7711cac3
I20240503 01:59:17.255220 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 49706355c8864e638e187eee7711cac3
I20240503 01:59:17.255285 398 log_verifier.cc:126] Checking tablet 5426821cad7e4b7b8689e5a6bfbad7ad
I20240503 01:59:17.263634 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 5426821cad7e4b7b8689e5a6bfbad7ad
I20240503 01:59:17.263736 398 log_verifier.cc:126] Checking tablet 5c289e9ad385459aa89fa32637a42f36
I20240503 01:59:17.269805 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 5c289e9ad385459aa89fa32637a42f36
I20240503 01:59:17.269886 398 log_verifier.cc:126] Checking tablet 5f821e4f2b37483abda9bd5a89cd0b19
I20240503 01:59:17.275516 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 5f821e4f2b37483abda9bd5a89cd0b19
I20240503 01:59:17.275588 398 log_verifier.cc:126] Checking tablet 7710730e9d5b440d98bd522c2f808fa3
I20240503 01:59:17.284018 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 7710730e9d5b440d98bd522c2f808fa3
I20240503 01:59:17.284103 398 log_verifier.cc:126] Checking tablet 7d729ec44ced4c01a4b84de1fcac3a6f
I20240503 01:59:17.289798 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7d729ec44ced4c01a4b84de1fcac3a6f
I20240503 01:59:17.289881 398 log_verifier.cc:126] Checking tablet 833d9ce859f84b499e667d86965d9fff
I20240503 01:59:17.295908 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 833d9ce859f84b499e667d86965d9fff
I20240503 01:59:17.295986 398 log_verifier.cc:126] Checking tablet 986d142f7be641ed9a9c338c41ba430a
I20240503 01:59:17.301904 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 986d142f7be641ed9a9c338c41ba430a
I20240503 01:59:17.301970 398 log_verifier.cc:126] Checking tablet a614380f92f64189ac5b5cdb42edf2e6
I20240503 01:59:17.307875 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet a614380f92f64189ac5b5cdb42edf2e6
I20240503 01:59:17.307956 398 log_verifier.cc:126] Checking tablet ae344bce9aeb452b848accc96ddbb1ac
I20240503 01:59:17.313796 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet ae344bce9aeb452b848accc96ddbb1ac
I20240503 01:59:17.313874 398 log_verifier.cc:126] Checking tablet c0bce60681a34a96a83284c762d6feee
I20240503 01:59:17.319790 398 log_verifier.cc:177] Verified matching terms for 6 ops in tablet c0bce60681a34a96a83284c762d6feee
I20240503 01:59:17.319864 398 log_verifier.cc:126] Checking tablet f586b7fd77824920b4300f38a29ba87b
I20240503 01:59:17.325742 398 log_verifier.cc:177] Verified matching terms for 3 ops in tablet f586b7fd77824920b4300f38a29ba87b
I20240503 01:59:17.326248 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 2317
I20240503 01:59:17.335359 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 2448
I20240503 01:59:17.343674 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 3452
I20240503 01:59:17.352699 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 3583
I20240503 01:59:17.360189 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 3714
I20240503 01:59:17.366701 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 3845
I20240503 01:59:17.372676 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 3976
I20240503 01:59:17.379727 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 5947
I20240503 01:59:17.386296 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 2226
2024-05-03T01:59:17Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/0 (27965 ms)
[----------] 1 test from RebalanceParamTest (27965 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/0
I20240503 01:59:17.429203 398 test_util.cc:268] Using random seed: 289010632
I20240503 01:59:17.430568 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 01:59:17.430640 398 ts_itest-base.cc:113] --------------
I20240503 01:59:17.430662 398 ts_itest-base.cc:114] 8 tablet servers
I20240503 01:59:17.430685 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 01:59:17.430708 398 ts_itest-base.cc:116] --------------
2024-05-03T01:59:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T01:59:17Z Disabled control of system clock
I20240503 01:59:17.440557 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:36303
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:36303
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240503 01:59:17.554961 6710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:17.555225 6710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:17.555275 6710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:17.559724 6710 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:59:17.559854 6710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 01:59:17.559895 6710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:17.559939 6710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 01:59:17.559973 6710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 01:59:17.566313 6710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:36303
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:36303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.6710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:17.567579 6710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:17.569191 6710 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:17.579442 6715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:17.580583 6716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:17.581534 6718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:17.582589 6710 server_base.cc:1034] running on GCE node
I20240503 01:59:17.583226 6710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:17.584503 6710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:17.587829 6710 hybrid_clock.cc:648] HybridClock initialized: now 1714701557587764 us; error 66 us; skew 500 ppm
I20240503 01:59:17.590679 6710 webserver.cc:460] Webserver started at http://127.0.99.190:39745/ using document root <none> and password file <none>
I20240503 01:59:17.591437 6710 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:17.591524 6710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:17.591751 6710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:17.593704 6710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "54aa976bc02b497e8f9115c0d4679ef6"
format_stamp: "Formatted at 2024-05-03 01:59:17 on dist-test-slave-w9tw"
I20240503 01:59:17.594288 6710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "54aa976bc02b497e8f9115c0d4679ef6"
format_stamp: "Formatted at 2024-05-03 01:59:17 on dist-test-slave-w9tw"
I20240503 01:59:17.598660 6710 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240503 01:59:17.601883 6724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:17.603389 6710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:17.603600 6710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "54aa976bc02b497e8f9115c0d4679ef6"
format_stamp: "Formatted at 2024-05-03 01:59:17 on dist-test-slave-w9tw"
I20240503 01:59:17.603765 6710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:17.624398 6710 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:17.625108 6710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:17.625272 6710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:17.632442 6710 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:36303
I20240503 01:59:17.632452 6776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:36303 every 8 connection(s)
I20240503 01:59:17.633505 6710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 01:59:17.636253 6777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:17.637207 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 6710
I20240503 01:59:17.637368 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 01:59:17.642230 6777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Bootstrap starting.
I20240503 01:59:17.644572 6777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:17.645444 6777 log.cc:824] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:17.647456 6777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: No bootstrap required, opened a new log
I20240503 01:59:17.650054 6777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54aa976bc02b497e8f9115c0d4679ef6" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36303 } }
I20240503 01:59:17.650278 6777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:17.650316 6777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54aa976bc02b497e8f9115c0d4679ef6, State: Initialized, Role: FOLLOWER
I20240503 01:59:17.650776 6777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54aa976bc02b497e8f9115c0d4679ef6" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36303 } }
I20240503 01:59:17.650905 6777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 01:59:17.650975 6777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 01:59:17.651082 6777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:17.651944 6777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54aa976bc02b497e8f9115c0d4679ef6" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36303 } }
I20240503 01:59:17.652330 6777 leader_election.cc:302] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54aa976bc02b497e8f9115c0d4679ef6; no voters:
I20240503 01:59:17.652652 6777 leader_election.cc:288] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 01:59:17.653019 6782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:17.653203 6782 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6 [term 1 LEADER]: Becoming Leader. State: Replica: 54aa976bc02b497e8f9115c0d4679ef6, State: Running, Role: LEADER
I20240503 01:59:17.653509 6782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54aa976bc02b497e8f9115c0d4679ef6" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36303 } }
I20240503 01:59:17.653822 6777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 01:59:17.655292 6783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54aa976bc02b497e8f9115c0d4679ef6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54aa976bc02b497e8f9115c0d4679ef6" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36303 } } }
I20240503 01:59:17.655426 6783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6 [sys.catalog]: This master's current role is: LEADER
I20240503 01:59:17.655773 6793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 01:59:17.656353 6784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54aa976bc02b497e8f9115c0d4679ef6. Latest consensus state: current_term: 1 leader_uuid: "54aa976bc02b497e8f9115c0d4679ef6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54aa976bc02b497e8f9115c0d4679ef6" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 36303 } } }
I20240503 01:59:17.656476 6784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6 [sys.catalog]: This master's current role is: LEADER
I20240503 01:59:17.658437 6793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 01:59:17.663734 6793 catalog_manager.cc:1347] Generated new cluster ID: 1c0cbc23992e45fa996243194f59455a
I20240503 01:59:17.663826 6793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 01:59:17.680162 6793 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 01:59:17.681504 6793 catalog_manager.cc:1504] Loading token signing keys...
I20240503 01:59:17.688983 6793 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Generated new TSK 0
I20240503 01:59:17.689893 6793 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 01:59:17.696715 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36303
--builtin_ntp_servers=127.0.99.148:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 01:59:17.810730 6801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:17.810992 6801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:17.811071 6801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:17.814304 6801 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:59:17.814424 6801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:17.814566 6801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 01:59:17.818349 6801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36303
--never_fsync=true
--heap_profile_path=/tmp/kudu.6801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:17.819317 6801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:17.820472 6801 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:17.829718 6806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:17.830622 6807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:17.831373 6809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:17.832623 6801 server_base.cc:1034] running on GCE node
I20240503 01:59:17.833060 6801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:17.833678 6801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:17.834826 6801 hybrid_clock.cc:648] HybridClock initialized: now 1714701557834813 us; error 35 us; skew 500 ppm
I20240503 01:59:17.836934 6801 webserver.cc:460] Webserver started at http://127.0.99.129:41983/ using document root <none> and password file <none>
I20240503 01:59:17.837529 6801 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:17.837596 6801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:17.837795 6801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:17.839561 6801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
format_stamp: "Formatted at 2024-05-03 01:59:17 on dist-test-slave-w9tw"
I20240503 01:59:17.840046 6801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
format_stamp: "Formatted at 2024-05-03 01:59:17 on dist-test-slave-w9tw"
I20240503 01:59:17.843771 6801 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 01:59:17.846151 6815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:17.847327 6801 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:17.847458 6801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
format_stamp: "Formatted at 2024-05-03 01:59:17 on dist-test-slave-w9tw"
I20240503 01:59:17.847563 6801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:17.869741 6801 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:17.870388 6801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:17.870610 6801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:17.871157 6801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:17.872068 6801 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:17.872119 6801 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:17.872181 6801 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:17.872207 6801 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:17.881496 6801 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:42943
I20240503 01:59:17.881527 6928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:42943 every 8 connection(s)
I20240503 01:59:17.882469 6801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 01:59:17.883010 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 6801
I20240503 01:59:17.883152 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 01:59:17.885566 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36303
--builtin_ntp_servers=127.0.99.148:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 01:59:17.893035 6929 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36303
I20240503 01:59:17.893296 6929 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:17.893798 6929 heartbeater.cc:511] Master 127.0.99.190:36303 requested a full tablet report, sending...
I20240503 01:59:17.895771 6741 ts_manager.cc:194] Registered new tserver with Master: 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:17.896994 6741 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:56821
W20240503 01:59:17.999095 6932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:17.999382 6932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:17.999456 6932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:18.002774 6932 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:59:18.002871 6932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:18.002977 6932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 01:59:18.006836 6932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36303
--never_fsync=true
--heap_profile_path=/tmp/kudu.6932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:18.007846 6932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:18.009004 6932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:18.018708 6937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:18.019579 6938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:18.020272 6940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:18.021504 6932 server_base.cc:1034] running on GCE node
I20240503 01:59:18.021981 6932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:18.022639 6932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:18.023805 6932 hybrid_clock.cc:648] HybridClock initialized: now 1714701558023797 us; error 31 us; skew 500 ppm
I20240503 01:59:18.025944 6932 webserver.cc:460] Webserver started at http://127.0.99.130:38611/ using document root <none> and password file <none>
I20240503 01:59:18.026553 6932 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:18.026619 6932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:18.026795 6932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:18.028528 6932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "63331f455be049c1bdd6c23c59ac1bef"
format_stamp: "Formatted at 2024-05-03 01:59:18 on dist-test-slave-w9tw"
I20240503 01:59:18.029026 6932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "63331f455be049c1bdd6c23c59ac1bef"
format_stamp: "Formatted at 2024-05-03 01:59:18 on dist-test-slave-w9tw"
I20240503 01:59:18.032647 6932 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:59:18.035204 6946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:18.036585 6932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:18.036731 6932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "63331f455be049c1bdd6c23c59ac1bef"
format_stamp: "Formatted at 2024-05-03 01:59:18 on dist-test-slave-w9tw"
I20240503 01:59:18.036844 6932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:18.069762 6932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:18.070399 6932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:18.070611 6932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:18.071154 6932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:18.072103 6932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:18.072154 6932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:18.072218 6932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:18.072243 6932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:18.081096 6932 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:37231
I20240503 01:59:18.081126 7059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:37231 every 8 connection(s)
I20240503 01:59:18.082115 6932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 01:59:18.091563 7060 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36303
I20240503 01:59:18.091794 7060 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:18.092051 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 6932
I20240503 01:59:18.092164 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 01:59:18.092312 7060 heartbeater.cc:511] Master 127.0.99.190:36303 requested a full tablet report, sending...
I20240503 01:59:18.093286 6740 ts_manager.cc:194] Registered new tserver with Master: 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:18.093931 6740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:43277
I20240503 01:59:18.094266 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36303
--builtin_ntp_servers=127.0.99.148:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 01:59:18.206972 7063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:18.207234 7063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:18.207281 7063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:18.210577 7063 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:59:18.210654 7063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:18.210757 7063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 01:59:18.214593 7063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36303
--never_fsync=true
--heap_profile_path=/tmp/kudu.7063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:18.215575 7063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:18.216711 7063 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:18.225914 7071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:18.226781 7069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:18.227593 7068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:18.228826 7063 server_base.cc:1034] running on GCE node
I20240503 01:59:18.229331 7063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:18.229956 7063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:18.231134 7063 hybrid_clock.cc:648] HybridClock initialized: now 1714701558231120 us; error 31 us; skew 500 ppm
I20240503 01:59:18.233208 7063 webserver.cc:460] Webserver started at http://127.0.99.131:43991/ using document root <none> and password file <none>
I20240503 01:59:18.233785 7063 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:18.233848 7063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:18.234023 7063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:18.235862 7063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
format_stamp: "Formatted at 2024-05-03 01:59:18 on dist-test-slave-w9tw"
I20240503 01:59:18.236558 7063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
format_stamp: "Formatted at 2024-05-03 01:59:18 on dist-test-slave-w9tw"
I20240503 01:59:18.240268 7063 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:18.242787 7077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:18.243911 7063 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:18.244030 7063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
format_stamp: "Formatted at 2024-05-03 01:59:18 on dist-test-slave-w9tw"
I20240503 01:59:18.244127 7063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:18.270282 7063 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:18.270951 7063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:18.271124 7063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:18.271682 7063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:18.272589 7063 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:18.272639 7063 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:18.272701 7063 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:18.272728 7063 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:18.281630 7063 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:41979
I20240503 01:59:18.281656 7190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:41979 every 8 connection(s)
I20240503 01:59:18.282621 7063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 01:59:18.290603 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 7063
I20240503 01:59:18.290740 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 01:59:18.292975 7191 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36303
I20240503 01:59:18.293264 7191 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:18.293691 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36303
--builtin_ntp_servers=127.0.99.148:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 01:59:18.293952 7191 heartbeater.cc:511] Master 127.0.99.190:36303 requested a full tablet report, sending...
I20240503 01:59:18.295212 6740 ts_manager.cc:194] Registered new tserver with Master: c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:18.295791 6740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:39795
W20240503 01:59:18.403791 7194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:18.404074 7194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:18.404148 7194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:18.407378 7194 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:59:18.407470 7194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:18.407567 7194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 01:59:18.411525 7194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36303
--never_fsync=true
--heap_profile_path=/tmp/kudu.7194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:18.412519 7194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:18.413636 7194 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:18.423301 7202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:18.424342 7200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:18.425247 7199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:18.426191 7194 server_base.cc:1034] running on GCE node
I20240503 01:59:18.426713 7194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:18.427362 7194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:18.428643 7194 hybrid_clock.cc:648] HybridClock initialized: now 1714701558428562 us; error 35 us; skew 500 ppm
I20240503 01:59:18.430969 7194 webserver.cc:460] Webserver started at http://127.0.99.132:40909/ using document root <none> and password file <none>
I20240503 01:59:18.431560 7194 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:18.431627 7194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:18.431826 7194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:18.433601 7194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "1d0e738c34af40fa9950c6f086619b3a"
format_stamp: "Formatted at 2024-05-03 01:59:18 on dist-test-slave-w9tw"
I20240503 01:59:18.434098 7194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "1d0e738c34af40fa9950c6f086619b3a"
format_stamp: "Formatted at 2024-05-03 01:59:18 on dist-test-slave-w9tw"
I20240503 01:59:18.438001 7194 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 01:59:18.440688 7208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:18.442018 7194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240503 01:59:18.442158 7194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "1d0e738c34af40fa9950c6f086619b3a"
format_stamp: "Formatted at 2024-05-03 01:59:18 on dist-test-slave-w9tw"
I20240503 01:59:18.442258 7194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:18.461601 7194 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:18.462262 7194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:18.462426 7194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:18.462994 7194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:18.463948 7194 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:18.463997 7194 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:18.464058 7194 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:18.464085 7194 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:18.473057 7194 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:36271
I20240503 01:59:18.473122 7321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:36271 every 8 connection(s)
I20240503 01:59:18.474043 7194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 01:59:18.480226 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 7194
I20240503 01:59:18.480350 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 01:59:18.482633 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36303
--builtin_ntp_servers=127.0.99.148:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 01:59:18.483659 7322 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36303
I20240503 01:59:18.483947 7322 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:18.484582 7322 heartbeater.cc:511] Master 127.0.99.190:36303 requested a full tablet report, sending...
I20240503 01:59:18.485803 6740 ts_manager.cc:194] Registered new tserver with Master: 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:18.486485 6740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:37377
W20240503 01:59:18.597157 7325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:18.597437 7325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:18.597486 7325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:18.600778 7325 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:59:18.600878 7325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:18.600956 7325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 01:59:18.605499 7325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36303
--never_fsync=true
--heap_profile_path=/tmp/kudu.7325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:18.606561 7325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:18.607726 7325 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:18.617666 7333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:18.618710 7331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:18.619879 7330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:18.620954 7325 server_base.cc:1034] running on GCE node
I20240503 01:59:18.621479 7325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:18.622102 7325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:18.623270 7325 hybrid_clock.cc:648] HybridClock initialized: now 1714701558623243 us; error 41 us; skew 500 ppm
I20240503 01:59:18.625396 7325 webserver.cc:460] Webserver started at http://127.0.99.133:37949/ using document root <none> and password file <none>
I20240503 01:59:18.626017 7325 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:18.626086 7325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:18.626287 7325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:18.628088 7325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "12e8c683db054df3b75c6285b98a5ad8"
format_stamp: "Formatted at 2024-05-03 01:59:18 on dist-test-slave-w9tw"
I20240503 01:59:18.628580 7325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "12e8c683db054df3b75c6285b98a5ad8"
format_stamp: "Formatted at 2024-05-03 01:59:18 on dist-test-slave-w9tw"
I20240503 01:59:18.632488 7325 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20240503 01:59:18.634987 7339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:18.636077 7325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:18.636202 7325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "12e8c683db054df3b75c6285b98a5ad8"
format_stamp: "Formatted at 2024-05-03 01:59:18 on dist-test-slave-w9tw"
I20240503 01:59:18.636304 7325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:18.651357 7325 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:18.651995 7325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:18.652159 7325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:18.652699 7325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:18.653605 7325 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:18.653656 7325 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:18.653697 7325 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:18.653728 7325 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:18.662338 7325 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:37387
I20240503 01:59:18.662396 7452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:37387 every 8 connection(s)
I20240503 01:59:18.663405 7325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 01:59:18.669801 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 7325
I20240503 01:59:18.669945 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 01:59:18.672201 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36303
--builtin_ntp_servers=127.0.99.148:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 01:59:18.672835 7453 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36303
I20240503 01:59:18.673118 7453 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:18.673615 7453 heartbeater.cc:511] Master 127.0.99.190:36303 requested a full tablet report, sending...
I20240503 01:59:18.674667 6740 ts_manager.cc:194] Registered new tserver with Master: 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:18.675482 6740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:35713
W20240503 01:59:18.789628 7456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:18.789868 7456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:18.789916 7456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:18.793439 7456 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:59:18.793541 7456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:18.793615 7456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 01:59:18.797614 7456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36303
--never_fsync=true
--heap_profile_path=/tmp/kudu.7456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:18.798647 7456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:18.799801 7456 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:18.809672 7461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:18.810249 7462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:18.811523 7464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:18.813026 7456 server_base.cc:1034] running on GCE node
I20240503 01:59:18.813546 7456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:18.814198 7456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:18.815402 7456 hybrid_clock.cc:648] HybridClock initialized: now 1714701558815356 us; error 66 us; skew 500 ppm
I20240503 01:59:18.817692 7456 webserver.cc:460] Webserver started at http://127.0.99.134:38403/ using document root <none> and password file <none>
I20240503 01:59:18.818346 7456 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:18.818409 7456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:18.818634 7456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:18.820447 7456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "bb5364d730cb4489a0bfc6c270b7450a"
format_stamp: "Formatted at 2024-05-03 01:59:18 on dist-test-slave-w9tw"
I20240503 01:59:18.820936 7456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "bb5364d730cb4489a0bfc6c270b7450a"
format_stamp: "Formatted at 2024-05-03 01:59:18 on dist-test-slave-w9tw"
I20240503 01:59:18.824616 7456 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 01:59:18.827836 7470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:18.829124 7456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 01:59:18.829272 7456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "bb5364d730cb4489a0bfc6c270b7450a"
format_stamp: "Formatted at 2024-05-03 01:59:18 on dist-test-slave-w9tw"
I20240503 01:59:18.829406 7456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:18.848656 7456 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:18.849346 7456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:18.849524 7456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:18.850054 7456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:18.851024 7456 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:18.851078 7456 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:18.851125 7456 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:18.851140 7456 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:18.860327 7456 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:43471
I20240503 01:59:18.860416 7583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:43471 every 8 connection(s)
I20240503 01:59:18.861335 7456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 01:59:18.869289 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 7456
I20240503 01:59:18.869426 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 01:59:18.870862 7584 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36303
I20240503 01:59:18.871146 7584 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:18.871678 7584 heartbeater.cc:511] Master 127.0.99.190:36303 requested a full tablet report, sending...
I20240503 01:59:18.871722 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36303
--builtin_ntp_servers=127.0.99.148:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 01:59:18.872746 6740 ts_manager.cc:194] Registered new tserver with Master: bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:18.873579 6740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:34741
I20240503 01:59:18.899499 6929 heartbeater.cc:503] Master 127.0.99.190:36303 was elected leader, sending a full tablet report...
W20240503 01:59:18.994068 7587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:18.994356 7587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:18.994434 7587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:18.997793 7587 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:59:18.997887 7587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:18.997990 7587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 01:59:19.001899 7587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36303
--never_fsync=true
--heap_profile_path=/tmp/kudu.7587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:19.003002 7587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:19.004209 7587 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:19.014096 7593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:19.014758 7595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:19.015776 7587 server_base.cc:1034] running on GCE node
W20240503 01:59:19.016867 7592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:19.017433 7587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:19.018078 7587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:19.019272 7587 hybrid_clock.cc:648] HybridClock initialized: now 1714701559019255 us; error 43 us; skew 500 ppm
I20240503 01:59:19.021596 7587 webserver.cc:460] Webserver started at http://127.0.99.135:41755/ using document root <none> and password file <none>
I20240503 01:59:19.022221 7587 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:19.022289 7587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:19.022462 7587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:19.024223 7587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
format_stamp: "Formatted at 2024-05-03 01:59:19 on dist-test-slave-w9tw"
I20240503 01:59:19.024703 7587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
format_stamp: "Formatted at 2024-05-03 01:59:19 on dist-test-slave-w9tw"
I20240503 01:59:19.028704 7587 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 01:59:19.031589 7601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:19.032826 7587 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20240503 01:59:19.032946 7587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
format_stamp: "Formatted at 2024-05-03 01:59:19 on dist-test-slave-w9tw"
I20240503 01:59:19.033061 7587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:19.083362 7587 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:19.084056 7587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:19.084228 7587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:19.084787 7587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:19.085701 7587 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:19.085755 7587 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:19.085799 7587 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:19.085824 7587 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:19.095283 7587 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:46087
I20240503 01:59:19.095350 7714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:46087 every 8 connection(s)
I20240503 01:59:19.096446 7587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 01:59:19.097090 7060 heartbeater.cc:503] Master 127.0.99.190:36303 was elected leader, sending a full tablet report...
I20240503 01:59:19.099401 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 7587
I20240503 01:59:19.099552 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 01:59:19.101850 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:0
--local_ip_for_outbound_sockets=127.0.99.136
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36303
--builtin_ntp_servers=127.0.99.148:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 01:59:19.107817 7715 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36303
I20240503 01:59:19.108258 7715 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:19.109063 7715 heartbeater.cc:511] Master 127.0.99.190:36303 requested a full tablet report, sending...
I20240503 01:59:19.110469 6740 ts_manager.cc:194] Registered new tserver with Master: b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:19.111125 6740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:41627
W20240503 01:59:19.224469 7718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:19.224735 7718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:19.224784 7718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:19.228389 7718 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:59:19.228474 7718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:19.228549 7718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 01:59:19.233542 7718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=0
--tserver_master_addrs=127.0.99.190:36303
--never_fsync=true
--heap_profile_path=/tmp/kudu.7718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:19.234616 7718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:19.235781 7718 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:19.246539 7726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:19.247618 7723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:19.248525 7724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:19.249584 7718 server_base.cc:1034] running on GCE node
I20240503 01:59:19.250066 7718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:19.250771 7718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:19.251960 7718 hybrid_clock.cc:648] HybridClock initialized: now 1714701559251935 us; error 45 us; skew 500 ppm
I20240503 01:59:19.254431 7718 webserver.cc:460] Webserver started at http://127.0.99.136:46033/ using document root <none> and password file <none>
I20240503 01:59:19.255170 7718 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:19.255239 7718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:19.255411 7718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 01:59:19.257249 7718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "db1a0000aaa346f79c0aa4b6c5105191"
format_stamp: "Formatted at 2024-05-03 01:59:19 on dist-test-slave-w9tw"
I20240503 01:59:19.257789 7718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "db1a0000aaa346f79c0aa4b6c5105191"
format_stamp: "Formatted at 2024-05-03 01:59:19 on dist-test-slave-w9tw"
I20240503 01:59:19.261559 7718 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 01:59:19.264176 7732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:19.265777 7718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:19.265939 7718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "db1a0000aaa346f79c0aa4b6c5105191"
format_stamp: "Formatted at 2024-05-03 01:59:19 on dist-test-slave-w9tw"
I20240503 01:59:19.266049 7718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:19.280920 7718 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:19.281662 7718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:19.281833 7718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:19.282397 7718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:19.283391 7718 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:19.283452 7718 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:19.283516 7718 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:19.283541 7718 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:19.292681 7718 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:42053
I20240503 01:59:19.292783 7845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:42053 every 8 connection(s)
I20240503 01:59:19.293779 7718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 01:59:19.298007 7191 heartbeater.cc:503] Master 127.0.99.190:36303 was elected leader, sending a full tablet report...
I20240503 01:59:19.298830 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 7718
I20240503 01:59:19.298974 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 01:59:19.304193 7846 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36303
I20240503 01:59:19.304446 7846 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:19.304956 7846 heartbeater.cc:511] Master 127.0.99.190:36303 requested a full tablet report, sending...
I20240503 01:59:19.305991 6740 ts_manager.cc:194] Registered new tserver with Master: db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:19.306785 6740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:56051
I20240503 01:59:19.314085 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 01:59:19.324576 398 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 01:59:19.324672 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 7325
I20240503 01:59:19.330298 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 7456
I20240503 01:59:19.336153 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 7587
I20240503 01:59:19.342252 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 7718
I20240503 01:59:19.488804 7322 heartbeater.cc:503] Master 127.0.99.190:36303 was elected leader, sending a full tablet report...
I20240503 01:59:23.100124 6740 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:23.122066 6994 tablet_service.cc:1467] Processing CreateTablet for tablet f56eecf7b234413aa854b77ef2094992 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f95424af8eee4c98883391a29cf46150]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:23.122159 7125 tablet_service.cc:1467] Processing CreateTablet for tablet f56eecf7b234413aa854b77ef2094992 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f95424af8eee4c98883391a29cf46150]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:23.122066 6993 tablet_service.cc:1467] Processing CreateTablet for tablet e4c02a8a79274ca1804b127579e28e7c (DEFAULT_TABLE table=rebalance_test_table_0 [id=f95424af8eee4c98883391a29cf46150]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:23.123212 6994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f56eecf7b234413aa854b77ef2094992. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:23.123212 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f56eecf7b234413aa854b77ef2094992. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:23.124869 6862 tablet_service.cc:1467] Processing CreateTablet for tablet 158f5b2278de4b4a82ee51924ee32c86 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f95424af8eee4c98883391a29cf46150]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:23.125566 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4c02a8a79274ca1804b127579e28e7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:23.125882 6862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 158f5b2278de4b4a82ee51924ee32c86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:23.122066 6992 tablet_service.cc:1467] Processing CreateTablet for tablet 158f5b2278de4b4a82ee51924ee32c86 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f95424af8eee4c98883391a29cf46150]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:23.126835 7124 tablet_service.cc:1467] Processing CreateTablet for tablet e4c02a8a79274ca1804b127579e28e7c (DEFAULT_TABLE table=rebalance_test_table_0 [id=f95424af8eee4c98883391a29cf46150]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:23.127084 6992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 158f5b2278de4b4a82ee51924ee32c86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:23.127084 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4c02a8a79274ca1804b127579e28e7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:23.124869 6863 tablet_service.cc:1467] Processing CreateTablet for tablet e4c02a8a79274ca1804b127579e28e7c (DEFAULT_TABLE table=rebalance_test_table_0 [id=f95424af8eee4c98883391a29cf46150]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:23.129354 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4c02a8a79274ca1804b127579e28e7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:23.133008 7866 tablet_bootstrap.cc:492] T 158f5b2278de4b4a82ee51924ee32c86 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:23.133073 7868 tablet_bootstrap.cc:492] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:23.133605 7867 tablet_bootstrap.cc:492] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:23.135758 7867 tablet_bootstrap.cc:654] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:23.135774 7866 tablet_bootstrap.cc:654] T 158f5b2278de4b4a82ee51924ee32c86 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:23.135591 7255 tablet_service.cc:1467] Processing CreateTablet for tablet 158f5b2278de4b4a82ee51924ee32c86 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f95424af8eee4c98883391a29cf46150]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:23.136649 7255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 158f5b2278de4b4a82ee51924ee32c86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:23.136610 7866 log.cc:824] T 158f5b2278de4b4a82ee51924ee32c86 P 63331f455be049c1bdd6c23c59ac1bef: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:23.136703 7868 tablet_bootstrap.cc:654] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:23.136880 7867 log.cc:824] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:23.137485 7256 tablet_service.cc:1467] Processing CreateTablet for tablet f56eecf7b234413aa854b77ef2094992 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f95424af8eee4c98883391a29cf46150]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:23.137555 7868 log.cc:824] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:23.137820 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f56eecf7b234413aa854b77ef2094992. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:23.139014 7867 tablet_bootstrap.cc:492] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:23.139186 7868 tablet_bootstrap.cc:492] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:23.139227 7867 ts_tablet_manager.cc:1367] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 01:59:23.139351 7866 tablet_bootstrap.cc:492] T 158f5b2278de4b4a82ee51924ee32c86 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:23.139375 7868 ts_tablet_manager.cc:1367] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 01:59:23.139598 7866 ts_tablet_manager.cc:1367] T 158f5b2278de4b4a82ee51924ee32c86 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20240503 01:59:23.143666 7866 raft_consensus.cc:348] T 158f5b2278de4b4a82ee51924ee32c86 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:23.144107 7866 raft_consensus.cc:374] T 158f5b2278de4b4a82ee51924ee32c86 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:23.144233 7866 raft_consensus.cc:729] T 158f5b2278de4b4a82ee51924ee32c86 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:23.144793 7866 consensus_queue.cc:260] T 158f5b2278de4b4a82ee51924ee32c86 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:23.145037 7868 raft_consensus.cc:348] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:23.145272 7868 raft_consensus.cc:374] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:23.145334 7868 raft_consensus.cc:729] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:23.145726 7867 raft_consensus.cc:348] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:23.145939 7867 raft_consensus.cc:374] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:23.145953 7866 ts_tablet_manager.cc:1398] T 158f5b2278de4b4a82ee51924ee32c86 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 01:59:23.146004 7867 raft_consensus.cc:729] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:23.146219 7866 tablet_bootstrap.cc:492] T e4c02a8a79274ca1804b127579e28e7c P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:23.146428 7867 consensus_queue.cc:260] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:23.147114 7868 consensus_queue.cc:260] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:23.147373 7866 tablet_bootstrap.cc:654] T e4c02a8a79274ca1804b127579e28e7c P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:23.147400 7867 ts_tablet_manager.cc:1398] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20240503 01:59:23.147784 7867 tablet_bootstrap.cc:492] T e4c02a8a79274ca1804b127579e28e7c P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:23.148874 7867 tablet_bootstrap.cc:654] T e4c02a8a79274ca1804b127579e28e7c P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:23.149354 7866 tablet_bootstrap.cc:492] T e4c02a8a79274ca1804b127579e28e7c P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:23.149461 7866 ts_tablet_manager.cc:1367] T e4c02a8a79274ca1804b127579e28e7c P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:23.149677 7868 ts_tablet_manager.cc:1398] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20240503 01:59:23.149972 7868 tablet_bootstrap.cc:492] T f56eecf7b234413aa854b77ef2094992 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:23.150081 7866 raft_consensus.cc:348] T e4c02a8a79274ca1804b127579e28e7c P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:23.150216 7866 raft_consensus.cc:374] T e4c02a8a79274ca1804b127579e28e7c P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:23.150257 7866 raft_consensus.cc:729] T e4c02a8a79274ca1804b127579e28e7c P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:23.150406 7866 consensus_queue.cc:260] T e4c02a8a79274ca1804b127579e28e7c P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:23.150753 7866 ts_tablet_manager.cc:1398] T e4c02a8a79274ca1804b127579e28e7c P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:23.150954 7866 tablet_bootstrap.cc:492] T f56eecf7b234413aa854b77ef2094992 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:23.151094 7868 tablet_bootstrap.cc:654] T f56eecf7b234413aa854b77ef2094992 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:23.152168 7872 tablet_bootstrap.cc:492] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:23.152346 7868 tablet_bootstrap.cc:492] T f56eecf7b234413aa854b77ef2094992 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:23.152441 7868 ts_tablet_manager.cc:1367] T f56eecf7b234413aa854b77ef2094992 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:23.152503 7866 tablet_bootstrap.cc:654] T f56eecf7b234413aa854b77ef2094992 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:23.153035 7868 raft_consensus.cc:348] T f56eecf7b234413aa854b77ef2094992 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:23.153182 7868 raft_consensus.cc:374] T f56eecf7b234413aa854b77ef2094992 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:23.153216 7868 raft_consensus.cc:729] T f56eecf7b234413aa854b77ef2094992 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:23.153362 7868 consensus_queue.cc:260] T f56eecf7b234413aa854b77ef2094992 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:23.153654 7868 ts_tablet_manager.cc:1398] T f56eecf7b234413aa854b77ef2094992 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:23.155119 7872 tablet_bootstrap.cc:654] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:23.156001 7866 tablet_bootstrap.cc:492] T f56eecf7b234413aa854b77ef2094992 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:23.156112 7866 ts_tablet_manager.cc:1367] T f56eecf7b234413aa854b77ef2094992 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 01:59:23.156687 7866 raft_consensus.cc:348] T f56eecf7b234413aa854b77ef2094992 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:23.156806 7866 raft_consensus.cc:374] T f56eecf7b234413aa854b77ef2094992 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:23.156857 7866 raft_consensus.cc:729] T f56eecf7b234413aa854b77ef2094992 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:23.157017 7866 consensus_queue.cc:260] T f56eecf7b234413aa854b77ef2094992 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:23.157377 7866 ts_tablet_manager.cc:1398] T f56eecf7b234413aa854b77ef2094992 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:23.158638 7867 tablet_bootstrap.cc:492] T e4c02a8a79274ca1804b127579e28e7c P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:23.158753 7867 ts_tablet_manager.cc:1367] T e4c02a8a79274ca1804b127579e28e7c P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 01:59:23.159408 7867 raft_consensus.cc:348] T e4c02a8a79274ca1804b127579e28e7c P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:23.159562 7867 raft_consensus.cc:374] T e4c02a8a79274ca1804b127579e28e7c P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:23.159598 7867 raft_consensus.cc:729] T e4c02a8a79274ca1804b127579e28e7c P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:23.159004 7872 log.cc:824] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:23.159739 7867 consensus_queue.cc:260] T e4c02a8a79274ca1804b127579e28e7c P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:23.160123 7867 ts_tablet_manager.cc:1398] T e4c02a8a79274ca1804b127579e28e7c P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:23.162593 7872 tablet_bootstrap.cc:492] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:23.162811 7872 ts_tablet_manager.cc:1367] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20240503 01:59:23.165796 7872 raft_consensus.cc:348] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:23.166023 7872 raft_consensus.cc:374] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:23.166079 7872 raft_consensus.cc:729] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:23.166486 7872 consensus_queue.cc:260] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:23.167423 7872 ts_tablet_manager.cc:1398] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 01:59:23.167693 7872 tablet_bootstrap.cc:492] T 158f5b2278de4b4a82ee51924ee32c86 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:23.168711 7872 tablet_bootstrap.cc:654] T 158f5b2278de4b4a82ee51924ee32c86 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:23.170243 7872 tablet_bootstrap.cc:492] T 158f5b2278de4b4a82ee51924ee32c86 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:23.170329 7872 ts_tablet_manager.cc:1367] T 158f5b2278de4b4a82ee51924ee32c86 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:23.170955 7872 raft_consensus.cc:348] T 158f5b2278de4b4a82ee51924ee32c86 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:23.171100 7872 raft_consensus.cc:374] T 158f5b2278de4b4a82ee51924ee32c86 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:23.171207 7872 raft_consensus.cc:729] T 158f5b2278de4b4a82ee51924ee32c86 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:23.171306 7872 consensus_queue.cc:260] T 158f5b2278de4b4a82ee51924ee32c86 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:23.171566 7872 ts_tablet_manager.cc:1398] T 158f5b2278de4b4a82ee51924ee32c86 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 01:59:23.229606 7323 tablet.cc:2369] T 158f5b2278de4b4a82ee51924ee32c86 P 1d0e738c34af40fa9950c6f086619b3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:23.254154 7881 raft_consensus.cc:482] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:23.254417 7881 raft_consensus.cc:504] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:23.255645 7881 leader_election.cc:288] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:23.259991 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56eecf7b234413aa854b77ef2094992" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:23.260216 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56eecf7b234413aa854b77ef2094992" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:23.260355 7145 raft_consensus.cc:2454] T f56eecf7b234413aa854b77ef2094992 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:23.260444 7014 raft_consensus.cc:2454] T f56eecf7b234413aa854b77ef2094992 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:23.260923 7211 leader_election.cc:302] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:23.261260 7881 raft_consensus.cc:2781] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:23.261341 7881 raft_consensus.cc:482] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:23.261387 7881 raft_consensus.cc:3037] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:23.264309 7881 raft_consensus.cc:504] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:23.264765 7881 leader_election.cc:288] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Requested vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:23.265339 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56eecf7b234413aa854b77ef2094992" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:23.265352 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56eecf7b234413aa854b77ef2094992" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:23.265487 7145 raft_consensus.cc:3037] T f56eecf7b234413aa854b77ef2094992 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:23.265507 7014 raft_consensus.cc:3037] T f56eecf7b234413aa854b77ef2094992 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:23.266804 7145 raft_consensus.cc:2454] T f56eecf7b234413aa854b77ef2094992 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:23.267135 7014 raft_consensus.cc:2454] T f56eecf7b234413aa854b77ef2094992 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:23.267241 7211 leader_election.cc:302] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:23.267460 7881 raft_consensus.cc:2781] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:23.267704 7881 raft_consensus.cc:686] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 01:59:23.268160 7881 consensus_queue.cc:237] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:23.271104 6739 catalog_manager.cc:5484] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:23.278836 7875 raft_consensus.cc:482] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:23.279000 7875 raft_consensus.cc:504] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:23.280079 7875 leader_election.cc:288] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:23.284634 6883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c02a8a79274ca1804b127579e28e7c" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:23.284719 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c02a8a79274ca1804b127579e28e7c" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:23.284844 7014 raft_consensus.cc:2454] T e4c02a8a79274ca1804b127579e28e7c P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:23.284976 6883 raft_consensus.cc:2454] T e4c02a8a79274ca1804b127579e28e7c P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:23.285218 7078 leader_election.cc:302] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:23.285481 7875 raft_consensus.cc:2781] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:23.285559 7875 raft_consensus.cc:482] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:23.285594 7875 raft_consensus.cc:3037] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:23.286666 7875 raft_consensus.cc:504] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:23.287106 7875 leader_election.cc:288] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Requested vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:23.287494 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c02a8a79274ca1804b127579e28e7c" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:23.287492 6883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c02a8a79274ca1804b127579e28e7c" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:23.287634 7014 raft_consensus.cc:3037] T e4c02a8a79274ca1804b127579e28e7c P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:23.287665 6883 raft_consensus.cc:3037] T e4c02a8a79274ca1804b127579e28e7c P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
W20240503 01:59:23.287724 7192 tablet.cc:2369] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:23.288673 7014 raft_consensus.cc:2454] T e4c02a8a79274ca1804b127579e28e7c P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:23.288673 6883 raft_consensus.cc:2454] T e4c02a8a79274ca1804b127579e28e7c P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:23.289014 7081 leader_election.cc:302] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:23.289259 7875 raft_consensus.cc:2781] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:23.289454 7875 raft_consensus.cc:686] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Becoming Leader. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Running, Role: LEADER
I20240503 01:59:23.289803 7875 consensus_queue.cc:237] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:23.293402 6741 catalog_manager.cc:5484] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
W20240503 01:59:23.337914 7061 tablet.cc:2369] T 158f5b2278de4b4a82ee51924ee32c86 P 63331f455be049c1bdd6c23c59ac1bef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:23.354928 7874 raft_consensus.cc:482] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:23.355078 7874 raft_consensus.cc:504] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:23.356472 7874 leader_election.cc:288] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:23.360759 7276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "158f5b2278de4b4a82ee51924ee32c86" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:23.360749 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "158f5b2278de4b4a82ee51924ee32c86" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:23.361127 7014 raft_consensus.cc:2454] T 158f5b2278de4b4a82ee51924ee32c86 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:23.361171 7276 raft_consensus.cc:2454] T 158f5b2278de4b4a82ee51924ee32c86 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:23.361599 6816 leader_election.cc:302] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:23.361876 7874 raft_consensus.cc:2781] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:23.361953 7874 raft_consensus.cc:482] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:23.361982 7874 raft_consensus.cc:3037] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:23.362941 7874 raft_consensus.cc:504] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:23.363345 7874 leader_election.cc:288] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:23.363809 7276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "158f5b2278de4b4a82ee51924ee32c86" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:23.363824 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "158f5b2278de4b4a82ee51924ee32c86" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:23.363932 7276 raft_consensus.cc:3037] T 158f5b2278de4b4a82ee51924ee32c86 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:23.363938 7014 raft_consensus.cc:3037] T 158f5b2278de4b4a82ee51924ee32c86 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:23.364681 7014 raft_consensus.cc:2454] T 158f5b2278de4b4a82ee51924ee32c86 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:23.365008 6816 leader_election.cc:302] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:23.365250 7874 raft_consensus.cc:2781] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:23.365254 7276 raft_consensus.cc:2454] T 158f5b2278de4b4a82ee51924ee32c86 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:23.365496 7874 raft_consensus.cc:686] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 01:59:23.365850 7874 consensus_queue.cc:237] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:23.368748 6741 catalog_manager.cc:5484] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
W20240503 01:59:23.392951 6930 tablet.cc:2369] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 01:59:23.529747 7895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:23.530042 7895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:23.533414 7895 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240503 01:59:23.650981 7145 raft_consensus.cc:1261] T f56eecf7b234413aa854b77ef2094992 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:23.651022 7014 raft_consensus.cc:1261] T f56eecf7b234413aa854b77ef2094992 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:23.652161 7014 raft_consensus.cc:1261] T e4c02a8a79274ca1804b127579e28e7c P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:23.652148 7885 consensus_queue.cc:1035] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:23.652669 7885 consensus_queue.cc:1035] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:23.652827 7875 consensus_queue.cc:1035] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:23.653805 6883 raft_consensus.cc:1261] T e4c02a8a79274ca1804b127579e28e7c P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:23.654600 7875 consensus_queue.cc:1035] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:23.666020 7012 raft_consensus.cc:1261] T 158f5b2278de4b4a82ee51924ee32c86 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:23.666340 7276 raft_consensus.cc:1261] T 158f5b2278de4b4a82ee51924ee32c86 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:23.666606 7893 consensus_queue.cc:1035] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:23.667078 7874 consensus_queue.cc:1035] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:23.673882 7922 mvcc.cc:205] Tried to move back new op lower bound from 7023417604733521920 to 7023417603237384192. Current Snapshot: MvccSnapshot[applied={T|T < 7023417604707352576}]
I20240503 01:59:23.674088 7925 mvcc.cc:205] Tried to move back new op lower bound from 7023417604707352576 to 7023417603237384192. Current Snapshot: MvccSnapshot[applied={T|T < 7023417604707352576}]
I20240503 01:59:23.676802 7921 mvcc.cc:205] Tried to move back new op lower bound from 7023417604706308096 to 7023417603148890112. Current Snapshot: MvccSnapshot[applied={T|T < 7023417604706308096}]
I20240503 01:59:23.679714 7928 mvcc.cc:205] Tried to move back new op lower bound from 7023417604706308096 to 7023417603148890112. Current Snapshot: MvccSnapshot[applied={T|T < 7023417604706308096}]
INSERT report
rows total: 2000
time total: 139.095 ms
time per row: 0.0695476 ms
I20240503 01:59:23.731613 6732 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:23.739094 7125 tablet_service.cc:1467] Processing CreateTablet for tablet ae9fb6c5833145ba91bbd7c24e68a7e2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38e214942900477190e2c3b67ccc5952]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:23.739482 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae9fb6c5833145ba91bbd7c24e68a7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:23.740255 6862 tablet_service.cc:1467] Processing CreateTablet for tablet ae9fb6c5833145ba91bbd7c24e68a7e2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38e214942900477190e2c3b67ccc5952]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:23.740239 7124 tablet_service.cc:1467] Processing CreateTablet for tablet 822bafcbc6de406997955ff7a1d9b612 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38e214942900477190e2c3b67ccc5952]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:23.740572 6862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae9fb6c5833145ba91bbd7c24e68a7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:23.740617 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 822bafcbc6de406997955ff7a1d9b612. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:23.742561 7952 tablet_bootstrap.cc:492] T ae9fb6c5833145ba91bbd7c24e68a7e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:23.743089 6863 tablet_service.cc:1467] Processing CreateTablet for tablet 822bafcbc6de406997955ff7a1d9b612 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38e214942900477190e2c3b67ccc5952]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:23.743155 6994 tablet_service.cc:1467] Processing CreateTablet for tablet 8c3d43c8b2ca42b19616fde1d7eb47a3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38e214942900477190e2c3b67ccc5952]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:23.743379 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 822bafcbc6de406997955ff7a1d9b612. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:23.743467 6994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c3d43c8b2ca42b19616fde1d7eb47a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:23.743650 7952 tablet_bootstrap.cc:654] T ae9fb6c5833145ba91bbd7c24e68a7e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:23.745050 7952 tablet_bootstrap.cc:492] T ae9fb6c5833145ba91bbd7c24e68a7e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:23.745147 7952 ts_tablet_manager.cc:1367] T ae9fb6c5833145ba91bbd7c24e68a7e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:23.745942 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 8c3d43c8b2ca42b19616fde1d7eb47a3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38e214942900477190e2c3b67ccc5952]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:23.746045 7953 tablet_bootstrap.cc:492] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:23.746237 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c3d43c8b2ca42b19616fde1d7eb47a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:23.746248 7952 raft_consensus.cc:348] T ae9fb6c5833145ba91bbd7c24e68a7e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:23.746387 7952 raft_consensus.cc:374] T ae9fb6c5833145ba91bbd7c24e68a7e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:23.746428 7952 raft_consensus.cc:729] T ae9fb6c5833145ba91bbd7c24e68a7e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:23.746673 7952 consensus_queue.cc:260] T ae9fb6c5833145ba91bbd7c24e68a7e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:23.747085 7952 ts_tablet_manager.cc:1398] T ae9fb6c5833145ba91bbd7c24e68a7e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:23.747157 7953 tablet_bootstrap.cc:654] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:23.747293 7952 tablet_bootstrap.cc:492] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:23.740239 7123 tablet_service.cc:1467] Processing CreateTablet for tablet 8c3d43c8b2ca42b19616fde1d7eb47a3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38e214942900477190e2c3b67ccc5952]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:23.749385 7952 tablet_bootstrap.cc:654] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:23.749436 7123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c3d43c8b2ca42b19616fde1d7eb47a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:23.749483 7256 tablet_service.cc:1467] Processing CreateTablet for tablet ae9fb6c5833145ba91bbd7c24e68a7e2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38e214942900477190e2c3b67ccc5952]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:23.749483 7255 tablet_service.cc:1467] Processing CreateTablet for tablet 822bafcbc6de406997955ff7a1d9b612 (DEFAULT_TABLE table=rebalance_test_table_1 [id=38e214942900477190e2c3b67ccc5952]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:23.749825 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae9fb6c5833145ba91bbd7c24e68a7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:23.749943 7255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 822bafcbc6de406997955ff7a1d9b612. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:23.751907 7952 tablet_bootstrap.cc:492] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:23.751994 7952 ts_tablet_manager.cc:1367] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240503 01:59:23.752619 7955 tablet_bootstrap.cc:492] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:23.752583 7952 raft_consensus.cc:348] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:23.752699 7952 raft_consensus.cc:374] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:23.752810 7952 raft_consensus.cc:729] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:23.752995 7952 consensus_queue.cc:260] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:23.753513 7952 ts_tablet_manager.cc:1398] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 01:59:23.753775 7952 tablet_bootstrap.cc:492] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:23.754071 7955 tablet_bootstrap.cc:654] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:23.755105 7952 tablet_bootstrap.cc:654] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:23.755416 7953 tablet_bootstrap.cc:492] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:23.755584 7953 ts_tablet_manager.cc:1367] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240503 01:59:23.755643 7955 tablet_bootstrap.cc:492] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:23.755760 7955 ts_tablet_manager.cc:1367] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:23.756399 7952 tablet_bootstrap.cc:492] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:23.756363 7955 raft_consensus.cc:348] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:23.756489 7952 ts_tablet_manager.cc:1367] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:23.756500 7955 raft_consensus.cc:374] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:23.756538 7955 raft_consensus.cc:729] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:23.756474 7953 raft_consensus.cc:348] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:23.756692 7953 raft_consensus.cc:374] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:23.756695 7955 consensus_queue.cc:260] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:23.756762 7953 raft_consensus.cc:729] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:23.756876 7958 tablet_bootstrap.cc:492] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:23.756982 7955 ts_tablet_manager.cc:1398] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:23.756994 7952 raft_consensus.cc:348] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:23.757010 7953 consensus_queue.cc:260] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:23.757113 7952 raft_consensus.cc:374] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:23.757154 7952 raft_consensus.cc:729] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:23.757294 7952 consensus_queue.cc:260] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:23.757476 7953 ts_tablet_manager.cc:1398] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:23.757752 7953 tablet_bootstrap.cc:492] T 822bafcbc6de406997955ff7a1d9b612 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:23.757830 7952 ts_tablet_manager.cc:1398] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:23.758846 7953 tablet_bootstrap.cc:654] T 822bafcbc6de406997955ff7a1d9b612 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:23.759121 7958 tablet_bootstrap.cc:654] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:23.760722 7953 tablet_bootstrap.cc:492] T 822bafcbc6de406997955ff7a1d9b612 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:23.760819 7953 ts_tablet_manager.cc:1367] T 822bafcbc6de406997955ff7a1d9b612 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:23.760329 7958 tablet_bootstrap.cc:492] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:23.761024 7958 ts_tablet_manager.cc:1367] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:23.761343 7953 raft_consensus.cc:348] T 822bafcbc6de406997955ff7a1d9b612 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:23.761479 7953 raft_consensus.cc:374] T 822bafcbc6de406997955ff7a1d9b612 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:23.761432 7958 raft_consensus.cc:348] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:23.761531 7953 raft_consensus.cc:729] T 822bafcbc6de406997955ff7a1d9b612 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:23.761543 7958 raft_consensus.cc:374] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:23.761581 7958 raft_consensus.cc:729] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:23.761667 7953 consensus_queue.cc:260] T 822bafcbc6de406997955ff7a1d9b612 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:23.761740 7958 consensus_queue.cc:260] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:23.761999 7953 ts_tablet_manager.cc:1398] T 822bafcbc6de406997955ff7a1d9b612 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:23.762153 7953 tablet_bootstrap.cc:492] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:23.762391 7958 ts_tablet_manager.cc:1398] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:23.762606 7958 tablet_bootstrap.cc:492] T 822bafcbc6de406997955ff7a1d9b612 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:23.763195 7953 tablet_bootstrap.cc:654] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:23.763546 7958 tablet_bootstrap.cc:654] T 822bafcbc6de406997955ff7a1d9b612 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:23.764555 7953 tablet_bootstrap.cc:492] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:23.764660 7953 ts_tablet_manager.cc:1367] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:59:23.765122 7958 tablet_bootstrap.cc:492] T 822bafcbc6de406997955ff7a1d9b612 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:23.765211 7958 ts_tablet_manager.cc:1367] T 822bafcbc6de406997955ff7a1d9b612 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:23.765286 7953 raft_consensus.cc:348] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:23.765417 7953 raft_consensus.cc:374] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:23.765451 7953 raft_consensus.cc:729] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:23.765600 7953 consensus_queue.cc:260] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:23.765702 7958 raft_consensus.cc:348] T 822bafcbc6de406997955ff7a1d9b612 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:23.765813 7958 raft_consensus.cc:374] T 822bafcbc6de406997955ff7a1d9b612 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:23.765866 7958 raft_consensus.cc:729] T 822bafcbc6de406997955ff7a1d9b612 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:23.765908 7953 ts_tablet_manager.cc:1398] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:23.766021 7958 consensus_queue.cc:260] T 822bafcbc6de406997955ff7a1d9b612 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:23.766309 7958 ts_tablet_manager.cc:1398] T 822bafcbc6de406997955ff7a1d9b612 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:23.770406 7875 raft_consensus.cc:482] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:23.770890 7873 raft_consensus.cc:482] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:23.770830 7875 raft_consensus.cc:504] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:23.771008 7873 raft_consensus.cc:504] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:23.771448 7875 leader_election.cc:288] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:23.772010 6883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "822bafcbc6de406997955ff7a1d9b612" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:23.772167 6883 raft_consensus.cc:2454] T 822bafcbc6de406997955ff7a1d9b612 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:23.772229 7873 leader_election.cc:288] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:23.773027 7081 leader_election.cc:302] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:23.773188 7875 raft_consensus.cc:2781] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:23.773231 7875 raft_consensus.cc:482] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:23.773255 7875 raft_consensus.cc:3037] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:23.774174 7875 raft_consensus.cc:504] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:23.774574 7875 leader_election.cc:288] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Requested vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:23.775605 6883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "822bafcbc6de406997955ff7a1d9b612" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:23.775743 6883 raft_consensus.cc:3037] T 822bafcbc6de406997955ff7a1d9b612 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:23.776424 7276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "822bafcbc6de406997955ff7a1d9b612" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:23.776580 7276 raft_consensus.cc:2454] T 822bafcbc6de406997955ff7a1d9b612 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:23.776926 6883 raft_consensus.cc:2454] T 822bafcbc6de406997955ff7a1d9b612 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:23.776969 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "822bafcbc6de406997955ff7a1d9b612" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:23.777182 7275 raft_consensus.cc:3037] T 822bafcbc6de406997955ff7a1d9b612 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:23.777551 7081 leader_election.cc:302] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:23.777937 7875 raft_consensus.cc:2781] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:23.777983 6883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3d43c8b2ca42b19616fde1d7eb47a3" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:23.778039 7875 raft_consensus.cc:686] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Becoming Leader. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Running, Role: LEADER
I20240503 01:59:23.778106 6883 raft_consensus.cc:2454] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:23.778191 7875 consensus_queue.cc:237] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:23.778491 6950 leader_election.cc:302] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:23.778633 7275 raft_consensus.cc:2454] T 822bafcbc6de406997955ff7a1d9b612 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:23.778800 7873 raft_consensus.cc:2781] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:23.778868 7873 raft_consensus.cc:482] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:23.778916 7873 raft_consensus.cc:3037] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:23.779954 7873 raft_consensus.cc:504] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:23.780355 7873 leader_election.cc:288] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:23.780272 6738 catalog_manager.cc:5484] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:23.780824 6883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3d43c8b2ca42b19616fde1d7eb47a3" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:23.780952 6883 raft_consensus.cc:3037] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:23.781419 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3d43c8b2ca42b19616fde1d7eb47a3" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:23.781476 7144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3d43c8b2ca42b19616fde1d7eb47a3" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:23.781543 7145 raft_consensus.cc:2454] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:23.781612 7144 raft_consensus.cc:3037] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:23.782495 7144 raft_consensus.cc:2454] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:23.782708 6883 raft_consensus.cc:2454] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:23.782995 6949 leader_election.cc:302] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:23.783214 7873 raft_consensus.cc:2781] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:23.783457 7873 raft_consensus.cc:686] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:23.783742 7873 consensus_queue.cc:237] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:23.785944 6739 catalog_manager.cc:5484] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:23.957955 7881 raft_consensus.cc:482] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:23.958114 7881 raft_consensus.cc:504] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:23.958746 7881 leader_election.cc:288] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:23.959506 7144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9fb6c5833145ba91bbd7c24e68a7e2" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:23.959766 7144 raft_consensus.cc:2454] T ae9fb6c5833145ba91bbd7c24e68a7e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:23.960141 7211 leader_election.cc:302] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:23.960350 7881 raft_consensus.cc:2781] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:23.960425 7881 raft_consensus.cc:482] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:23.960454 7881 raft_consensus.cc:3037] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:23.961284 7881 raft_consensus.cc:504] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:23.961706 7881 leader_election.cc:288] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Requested vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:23.962220 7144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9fb6c5833145ba91bbd7c24e68a7e2" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:23.962373 7144 raft_consensus.cc:3037] T ae9fb6c5833145ba91bbd7c24e68a7e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:23.963335 7144 raft_consensus.cc:2454] T ae9fb6c5833145ba91bbd7c24e68a7e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:23.963678 7211 leader_election.cc:302] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:23.963881 7881 raft_consensus.cc:2781] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:23.963970 7881 raft_consensus.cc:686] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 01:59:23.964130 7881 consensus_queue.cc:237] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:23.964728 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9fb6c5833145ba91bbd7c24e68a7e2" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:23.964893 6882 raft_consensus.cc:3037] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:23.965921 6882 raft_consensus.cc:2454] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:23.965930 6739 catalog_manager.cc:5484] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:23.966184 6883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae9fb6c5833145ba91bbd7c24e68a7e2" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:23.966329 6883 raft_consensus.cc:2362] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1. Re-sending same reply.
W20240503 01:59:24.109727 7972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:24.110023 7972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:24.113477 7972 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 01:59:24.165167 7873 consensus_queue.cc:1035] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:24.169306 7968 consensus_queue.cc:1035] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:24.212194 7275 raft_consensus.cc:1261] T 822bafcbc6de406997955ff7a1d9b612 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:24.212477 6883 raft_consensus.cc:1261] T 822bafcbc6de406997955ff7a1d9b612 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:24.212874 7875 consensus_queue.cc:1035] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:24.213145 7889 consensus_queue.cc:1035] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:24.216413 6881 raft_consensus.cc:1261] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:24.216526 7145 raft_consensus.cc:1261] T ae9fb6c5833145ba91bbd7c24e68a7e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:24.216905 7881 consensus_queue.cc:1035] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:24.219367 7881 consensus_queue.cc:1035] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 124.826 ms
time per row: 0.0624132 ms
I20240503 01:59:24.291606 6741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:24.298408 7255 tablet_service.cc:1467] Processing CreateTablet for tablet d9896016473544858f54bc801b689a05 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ca54251151134e5998e64c31c165357e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:24.298616 7256 tablet_service.cc:1467] Processing CreateTablet for tablet b510e069ae4c4e86ab35c6f43f9bbdfe (DEFAULT_TABLE table=rebalance_test_table_2 [id=ca54251151134e5998e64c31c165357e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:24.298807 7255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9896016473544858f54bc801b689a05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:24.299127 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b510e069ae4c4e86ab35c6f43f9bbdfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:24.299276 6861 tablet_service.cc:1467] Processing CreateTablet for tablet d9896016473544858f54bc801b689a05 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ca54251151134e5998e64c31c165357e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:24.299535 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9896016473544858f54bc801b689a05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:24.301441 8021 tablet_bootstrap.cc:492] T d9896016473544858f54bc801b689a05 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:24.302078 6863 tablet_service.cc:1467] Processing CreateTablet for tablet 5de30ce4152042bba22b892be1e61712 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ca54251151134e5998e64c31c165357e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:24.302347 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5de30ce4152042bba22b892be1e61712. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:24.302367 6994 tablet_service.cc:1467] Processing CreateTablet for tablet d9896016473544858f54bc801b689a05 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ca54251151134e5998e64c31c165357e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:24.302786 8021 tablet_bootstrap.cc:654] T d9896016473544858f54bc801b689a05 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:24.302827 6994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9896016473544858f54bc801b689a05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:24.304494 7254 tablet_service.cc:1467] Processing CreateTablet for tablet 5de30ce4152042bba22b892be1e61712 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ca54251151134e5998e64c31c165357e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:24.304780 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5de30ce4152042bba22b892be1e61712. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:24.305431 8024 tablet_bootstrap.cc:492] T d9896016473544858f54bc801b689a05 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:24.305550 6993 tablet_service.cc:1467] Processing CreateTablet for tablet b510e069ae4c4e86ab35c6f43f9bbdfe (DEFAULT_TABLE table=rebalance_test_table_2 [id=ca54251151134e5998e64c31c165357e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:24.305804 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b510e069ae4c4e86ab35c6f43f9bbdfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:24.306639 8024 tablet_bootstrap.cc:654] T d9896016473544858f54bc801b689a05 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:24.308056 6992 tablet_service.cc:1467] Processing CreateTablet for tablet 5de30ce4152042bba22b892be1e61712 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ca54251151134e5998e64c31c165357e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:24.308265 8021 tablet_bootstrap.cc:492] T d9896016473544858f54bc801b689a05 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:24.308292 6992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5de30ce4152042bba22b892be1e61712. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:24.308372 8021 ts_tablet_manager.cc:1367] T d9896016473544858f54bc801b689a05 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:59:24.308813 8021 raft_consensus.cc:348] T d9896016473544858f54bc801b689a05 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:24.308921 8021 raft_consensus.cc:374] T d9896016473544858f54bc801b689a05 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:24.308961 8021 raft_consensus.cc:729] T d9896016473544858f54bc801b689a05 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:24.309129 8021 consensus_queue.cc:260] T d9896016473544858f54bc801b689a05 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:24.309429 8021 ts_tablet_manager.cc:1398] T d9896016473544858f54bc801b689a05 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:24.309625 8021 tablet_bootstrap.cc:492] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:24.310289 8024 tablet_bootstrap.cc:492] T d9896016473544858f54bc801b689a05 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:24.310401 8024 ts_tablet_manager.cc:1367] T d9896016473544858f54bc801b689a05 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 01:59:24.310709 8021 tablet_bootstrap.cc:654] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:24.311102 8024 raft_consensus.cc:348] T d9896016473544858f54bc801b689a05 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:24.311228 8024 raft_consensus.cc:374] T d9896016473544858f54bc801b689a05 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:24.311265 8024 raft_consensus.cc:729] T d9896016473544858f54bc801b689a05 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:24.311417 8024 consensus_queue.cc:260] T d9896016473544858f54bc801b689a05 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:24.311883 8021 tablet_bootstrap.cc:492] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:24.311954 8021 ts_tablet_manager.cc:1367] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:24.312115 7123 tablet_service.cc:1467] Processing CreateTablet for tablet b510e069ae4c4e86ab35c6f43f9bbdfe (DEFAULT_TABLE table=rebalance_test_table_2 [id=ca54251151134e5998e64c31c165357e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:24.312424 7123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b510e069ae4c4e86ab35c6f43f9bbdfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:24.312397 8021 raft_consensus.cc:348] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:24.312497 8021 raft_consensus.cc:374] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:24.312542 8021 raft_consensus.cc:729] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:24.311729 8024 ts_tablet_manager.cc:1398] T d9896016473544858f54bc801b689a05 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:24.312672 8021 consensus_queue.cc:260] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:24.312803 8024 tablet_bootstrap.cc:492] T b510e069ae4c4e86ab35c6f43f9bbdfe P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:24.312959 8021 ts_tablet_manager.cc:1398] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:24.313114 8021 tablet_bootstrap.cc:492] T 5de30ce4152042bba22b892be1e61712 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:24.314051 8021 tablet_bootstrap.cc:654] T 5de30ce4152042bba22b892be1e61712 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:24.314044 8024 tablet_bootstrap.cc:654] T b510e069ae4c4e86ab35c6f43f9bbdfe P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:24.315420 8021 tablet_bootstrap.cc:492] T 5de30ce4152042bba22b892be1e61712 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:24.315526 8021 ts_tablet_manager.cc:1367] T 5de30ce4152042bba22b892be1e61712 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:24.315989 8021 raft_consensus.cc:348] T 5de30ce4152042bba22b892be1e61712 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:24.316102 8021 raft_consensus.cc:374] T 5de30ce4152042bba22b892be1e61712 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:24.316133 8021 raft_consensus.cc:729] T 5de30ce4152042bba22b892be1e61712 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:24.316299 8021 consensus_queue.cc:260] T 5de30ce4152042bba22b892be1e61712 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:24.316676 8021 ts_tablet_manager.cc:1398] T 5de30ce4152042bba22b892be1e61712 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:24.316789 8029 tablet_bootstrap.cc:492] T b510e069ae4c4e86ab35c6f43f9bbdfe P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:24.317893 8029 tablet_bootstrap.cc:654] T b510e069ae4c4e86ab35c6f43f9bbdfe P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:24.318981 8022 tablet_bootstrap.cc:492] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:24.319514 8029 tablet_bootstrap.cc:492] T b510e069ae4c4e86ab35c6f43f9bbdfe P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:24.319619 8029 ts_tablet_manager.cc:1367] T b510e069ae4c4e86ab35c6f43f9bbdfe P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:24.319929 8022 tablet_bootstrap.cc:654] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:24.320080 8029 raft_consensus.cc:348] T b510e069ae4c4e86ab35c6f43f9bbdfe P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:24.320184 8029 raft_consensus.cc:374] T b510e069ae4c4e86ab35c6f43f9bbdfe P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:24.320230 8029 raft_consensus.cc:729] T b510e069ae4c4e86ab35c6f43f9bbdfe P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:24.320358 8029 consensus_queue.cc:260] T b510e069ae4c4e86ab35c6f43f9bbdfe P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:24.320746 8029 ts_tablet_manager.cc:1398] T b510e069ae4c4e86ab35c6f43f9bbdfe P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:24.321098 8022 tablet_bootstrap.cc:492] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:24.321193 8022 ts_tablet_manager.cc:1367] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:24.321404 8024 tablet_bootstrap.cc:492] T b510e069ae4c4e86ab35c6f43f9bbdfe P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:24.321621 8024 ts_tablet_manager.cc:1367] T b510e069ae4c4e86ab35c6f43f9bbdfe P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240503 01:59:24.321820 8022 raft_consensus.cc:348] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:24.321940 8022 raft_consensus.cc:374] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:24.321981 8022 raft_consensus.cc:729] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:24.322122 8022 consensus_queue.cc:260] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:24.322172 8024 raft_consensus.cc:348] T b510e069ae4c4e86ab35c6f43f9bbdfe P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:24.322278 8024 raft_consensus.cc:374] T b510e069ae4c4e86ab35c6f43f9bbdfe P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:24.322321 8024 raft_consensus.cc:729] T b510e069ae4c4e86ab35c6f43f9bbdfe P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:24.322432 8022 ts_tablet_manager.cc:1398] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:24.322451 8024 consensus_queue.cc:260] T b510e069ae4c4e86ab35c6f43f9bbdfe P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:24.322659 8022 tablet_bootstrap.cc:492] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:24.322757 8024 ts_tablet_manager.cc:1398] T b510e069ae4c4e86ab35c6f43f9bbdfe P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:24.322917 8024 tablet_bootstrap.cc:492] T 5de30ce4152042bba22b892be1e61712 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:24.324003 8024 tablet_bootstrap.cc:654] T 5de30ce4152042bba22b892be1e61712 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:24.324606 8022 tablet_bootstrap.cc:654] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:24.325955 8022 tablet_bootstrap.cc:492] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:24.326043 8022 ts_tablet_manager.cc:1367] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:24.326488 8022 raft_consensus.cc:348] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:24.326651 8022 raft_consensus.cc:374] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:24.326656 8024 tablet_bootstrap.cc:492] T 5de30ce4152042bba22b892be1e61712 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:24.326679 8022 raft_consensus.cc:729] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:24.326761 8024 ts_tablet_manager.cc:1367] T 5de30ce4152042bba22b892be1e61712 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 01:59:24.326846 8022 consensus_queue.cc:260] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:24.327131 8022 ts_tablet_manager.cc:1398] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:24.327230 8024 raft_consensus.cc:348] T 5de30ce4152042bba22b892be1e61712 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:24.327345 8024 raft_consensus.cc:374] T 5de30ce4152042bba22b892be1e61712 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:24.327386 8024 raft_consensus.cc:729] T 5de30ce4152042bba22b892be1e61712 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:24.327512 8024 consensus_queue.cc:260] T 5de30ce4152042bba22b892be1e61712 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:24.327797 8024 ts_tablet_manager.cc:1398] T 5de30ce4152042bba22b892be1e61712 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:24.340701 7874 raft_consensus.cc:482] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:24.340865 7874 raft_consensus.cc:504] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:24.341317 7874 leader_election.cc:288] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:24.341801 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9896016473544858f54bc801b689a05" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:24.341807 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9896016473544858f54bc801b689a05" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:24.341948 7012 raft_consensus.cc:2454] T d9896016473544858f54bc801b689a05 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:24.341950 7275 raft_consensus.cc:2454] T d9896016473544858f54bc801b689a05 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:24.342293 6817 leader_election.cc:302] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:24.342490 7874 raft_consensus.cc:2781] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:24.342572 7874 raft_consensus.cc:482] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:24.342618 7874 raft_consensus.cc:3037] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:24.343456 7874 raft_consensus.cc:504] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:24.343876 7874 leader_election.cc:288] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:24.344230 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9896016473544858f54bc801b689a05" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:24.344271 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9896016473544858f54bc801b689a05" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:24.344385 7275 raft_consensus.cc:3037] T d9896016473544858f54bc801b689a05 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:24.344394 7012 raft_consensus.cc:3037] T d9896016473544858f54bc801b689a05 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:24.345237 7275 raft_consensus.cc:2454] T d9896016473544858f54bc801b689a05 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:24.345309 7012 raft_consensus.cc:2454] T d9896016473544858f54bc801b689a05 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:24.345607 6816 leader_election.cc:302] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:24.345846 7874 raft_consensus.cc:2781] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:24.345980 7874 raft_consensus.cc:686] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 01:59:24.346113 7874 consensus_queue.cc:237] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:24.348042 6741 catalog_manager.cc:5484] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:24.351482 8013 raft_consensus.cc:482] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:24.351626 8013 raft_consensus.cc:504] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:24.352066 8013 leader_election.cc:288] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:24.352618 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b510e069ae4c4e86ab35c6f43f9bbdfe" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:24.352618 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b510e069ae4c4e86ab35c6f43f9bbdfe" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:24.352777 7012 raft_consensus.cc:2454] T b510e069ae4c4e86ab35c6f43f9bbdfe P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:24.352777 7145 raft_consensus.cc:2454] T b510e069ae4c4e86ab35c6f43f9bbdfe P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:24.353211 7211 leader_election.cc:302] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:24.353518 8013 raft_consensus.cc:2781] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:24.353578 8013 raft_consensus.cc:482] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:24.353607 8013 raft_consensus.cc:3037] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:24.355067 8013 raft_consensus.cc:504] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:24.355504 8013 leader_election.cc:288] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Requested vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:24.356086 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b510e069ae4c4e86ab35c6f43f9bbdfe" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:24.356084 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b510e069ae4c4e86ab35c6f43f9bbdfe" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:24.356333 7012 raft_consensus.cc:3037] T b510e069ae4c4e86ab35c6f43f9bbdfe P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:24.356334 7145 raft_consensus.cc:3037] T b510e069ae4c4e86ab35c6f43f9bbdfe P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:24.357638 7012 raft_consensus.cc:2454] T b510e069ae4c4e86ab35c6f43f9bbdfe P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:24.357652 7145 raft_consensus.cc:2454] T b510e069ae4c4e86ab35c6f43f9bbdfe P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:24.357976 7211 leader_election.cc:302] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:24.358125 8013 raft_consensus.cc:2781] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:24.358214 8013 raft_consensus.cc:686] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 01:59:24.358350 8013 consensus_queue.cc:237] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:24.360843 6732 catalog_manager.cc:5484] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:24.573913 7874 raft_consensus.cc:482] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:24.574080 7874 raft_consensus.cc:504] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:24.574509 7874 leader_election.cc:288] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:24.575062 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5de30ce4152042bba22b892be1e61712" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:24.575063 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5de30ce4152042bba22b892be1e61712" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:24.575196 7275 raft_consensus.cc:2454] T 5de30ce4152042bba22b892be1e61712 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:24.575234 7012 raft_consensus.cc:2454] T 5de30ce4152042bba22b892be1e61712 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:24.575570 6816 leader_election.cc:302] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:24.575742 7874 raft_consensus.cc:2781] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:24.575801 7874 raft_consensus.cc:482] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:24.575843 7874 raft_consensus.cc:3037] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:24.577024 7874 raft_consensus.cc:504] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:24.577672 7874 leader_election.cc:288] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Requested vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:24.578125 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5de30ce4152042bba22b892be1e61712" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:24.578142 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5de30ce4152042bba22b892be1e61712" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:24.578253 7275 raft_consensus.cc:3037] T 5de30ce4152042bba22b892be1e61712 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:24.578259 7012 raft_consensus.cc:3037] T 5de30ce4152042bba22b892be1e61712 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:24.579150 7275 raft_consensus.cc:2454] T 5de30ce4152042bba22b892be1e61712 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:24.579151 7012 raft_consensus.cc:2454] T 5de30ce4152042bba22b892be1e61712 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:24.579490 6817 leader_election.cc:302] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:24.579679 7874 raft_consensus.cc:2781] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:24.579885 7874 raft_consensus.cc:686] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 01:59:24.580104 7874 consensus_queue.cc:237] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:24.582002 6732 catalog_manager.cc:5484] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
W20240503 01:59:24.733901 8038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:24.734185 8038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:24.737558 8038 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 01:59:24.822250 8013 consensus_queue.cc:1035] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:24.826727 8013 consensus_queue.cc:1035] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:24.837729 7874 consensus_queue.cc:1035] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:24.842880 7874 consensus_queue.cc:1035] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:24.853407 7014 raft_consensus.cc:1261] T 5de30ce4152042bba22b892be1e61712 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:24.853878 8034 consensus_queue.cc:1035] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:24.864933 7275 raft_consensus.cc:1261] T 5de30ce4152042bba22b892be1e61712 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:24.865422 8034 consensus_queue.cc:1035] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 119.19 ms
time per row: 0.059595 ms
I20240503 01:59:24.909089 6738 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:24.916764 7254 tablet_service.cc:1467] Processing CreateTablet for tablet 4bd8226227c348d5a5d4d8ff28959d27 (DEFAULT_TABLE table=rebalance_test_table_3 [id=45e94c90bda847688db4572a4f9d0c02]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:24.917114 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bd8226227c348d5a5d4d8ff28959d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:24.917497 6992 tablet_service.cc:1467] Processing CreateTablet for tablet 8ed2793842464443826f83f60fb6dcd1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=45e94c90bda847688db4572a4f9d0c02]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:24.917532 6993 tablet_service.cc:1467] Processing CreateTablet for tablet a5e23eebc629494cbb8262bc123a720a (DEFAULT_TABLE table=rebalance_test_table_3 [id=45e94c90bda847688db4572a4f9d0c02]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:24.917789 6992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ed2793842464443826f83f60fb6dcd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:24.918820 7256 tablet_service.cc:1467] Processing CreateTablet for tablet a5e23eebc629494cbb8262bc123a720a (DEFAULT_TABLE table=rebalance_test_table_3 [id=45e94c90bda847688db4572a4f9d0c02]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:24.918870 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5e23eebc629494cbb8262bc123a720a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:24.919090 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5e23eebc629494cbb8262bc123a720a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:24.921456 7123 tablet_service.cc:1467] Processing CreateTablet for tablet 8ed2793842464443826f83f60fb6dcd1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=45e94c90bda847688db4572a4f9d0c02]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:24.921533 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 4bd8226227c348d5a5d4d8ff28959d27 (DEFAULT_TABLE table=rebalance_test_table_3 [id=45e94c90bda847688db4572a4f9d0c02]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:24.921527 7125 tablet_service.cc:1467] Processing CreateTablet for tablet a5e23eebc629494cbb8262bc123a720a (DEFAULT_TABLE table=rebalance_test_table_3 [id=45e94c90bda847688db4572a4f9d0c02]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:24.921751 7123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ed2793842464443826f83f60fb6dcd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:24.921767 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bd8226227c348d5a5d4d8ff28959d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:24.921897 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5e23eebc629494cbb8262bc123a720a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:24.921890 8097 tablet_bootstrap.cc:492] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:24.922821 8097 tablet_bootstrap.cc:654] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:24.924407 8099 tablet_bootstrap.cc:492] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:24.924538 6863 tablet_service.cc:1467] Processing CreateTablet for tablet 8ed2793842464443826f83f60fb6dcd1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=45e94c90bda847688db4572a4f9d0c02]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:24.924839 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ed2793842464443826f83f60fb6dcd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:24.925089 8099 tablet_bootstrap.cc:654] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:24.925880 8100 tablet_bootstrap.cc:492] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:24.926625 8097 tablet_bootstrap.cc:492] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:24.926716 8097 ts_tablet_manager.cc:1367] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:24.927217 7124 tablet_service.cc:1467] Processing CreateTablet for tablet 4bd8226227c348d5a5d4d8ff28959d27 (DEFAULT_TABLE table=rebalance_test_table_3 [id=45e94c90bda847688db4572a4f9d0c02]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:24.927225 8097 raft_consensus.cc:348] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:24.927326 8097 raft_consensus.cc:374] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:24.927351 8097 raft_consensus.cc:729] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:24.927438 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bd8226227c348d5a5d4d8ff28959d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:24.927481 8097 consensus_queue.cc:260] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:24.927747 8097 ts_tablet_manager.cc:1398] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:24.928169 8099 tablet_bootstrap.cc:492] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:24.928256 8099 ts_tablet_manager.cc:1367] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:24.928697 8099 raft_consensus.cc:348] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:24.928807 8099 raft_consensus.cc:374] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:24.928848 8099 raft_consensus.cc:729] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:24.928977 8099 consensus_queue.cc:260] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:24.929551 8099 ts_tablet_manager.cc:1398] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:24.929740 8099 tablet_bootstrap.cc:492] T 8ed2793842464443826f83f60fb6dcd1 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:24.930454 8096 tablet_bootstrap.cc:492] T 8ed2793842464443826f83f60fb6dcd1 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:24.931234 8099 tablet_bootstrap.cc:654] T 8ed2793842464443826f83f60fb6dcd1 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:24.931537 8096 tablet_bootstrap.cc:654] T 8ed2793842464443826f83f60fb6dcd1 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:24.932133 8097 tablet_bootstrap.cc:492] T 4bd8226227c348d5a5d4d8ff28959d27 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:24.932507 8100 tablet_bootstrap.cc:654] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:24.933065 8097 tablet_bootstrap.cc:654] T 4bd8226227c348d5a5d4d8ff28959d27 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:24.933460 8100 tablet_bootstrap.cc:492] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:24.933670 8100 ts_tablet_manager.cc:1367] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 01:59:24.934192 8099 tablet_bootstrap.cc:492] T 8ed2793842464443826f83f60fb6dcd1 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:24.934195 8097 tablet_bootstrap.cc:492] T 4bd8226227c348d5a5d4d8ff28959d27 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:24.934278 8097 ts_tablet_manager.cc:1367] T 4bd8226227c348d5a5d4d8ff28959d27 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:24.934278 8099 ts_tablet_manager.cc:1367] T 8ed2793842464443826f83f60fb6dcd1 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:24.934295 8096 tablet_bootstrap.cc:492] T 8ed2793842464443826f83f60fb6dcd1 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:24.934371 8096 ts_tablet_manager.cc:1367] T 8ed2793842464443826f83f60fb6dcd1 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 01:59:24.934779 8099 raft_consensus.cc:348] T 8ed2793842464443826f83f60fb6dcd1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:24.934824 8096 raft_consensus.cc:348] T 8ed2793842464443826f83f60fb6dcd1 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:24.934805 8097 raft_consensus.cc:348] T 4bd8226227c348d5a5d4d8ff28959d27 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:24.934913 8099 raft_consensus.cc:374] T 8ed2793842464443826f83f60fb6dcd1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:24.934913 8096 raft_consensus.cc:374] T 8ed2793842464443826f83f60fb6dcd1 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:24.934914 8097 raft_consensus.cc:374] T 4bd8226227c348d5a5d4d8ff28959d27 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:24.934949 8097 raft_consensus.cc:729] T 4bd8226227c348d5a5d4d8ff28959d27 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:24.934947 8096 raft_consensus.cc:729] T 8ed2793842464443826f83f60fb6dcd1 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:24.934947 8099 raft_consensus.cc:729] T 8ed2793842464443826f83f60fb6dcd1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:24.935040 8100 raft_consensus.cc:348] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:24.935078 8097 consensus_queue.cc:260] T 4bd8226227c348d5a5d4d8ff28959d27 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:24.935081 8096 consensus_queue.cc:260] T 8ed2793842464443826f83f60fb6dcd1 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:24.935081 8099 consensus_queue.cc:260] T 8ed2793842464443826f83f60fb6dcd1 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:24.935135 8100 raft_consensus.cc:374] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:24.935166 8100 raft_consensus.cc:729] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:24.935348 8096 ts_tablet_manager.cc:1398] T 8ed2793842464443826f83f60fb6dcd1 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:24.935348 8099 ts_tablet_manager.cc:1398] T 8ed2793842464443826f83f60fb6dcd1 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:24.935355 8097 ts_tablet_manager.cc:1398] T 4bd8226227c348d5a5d4d8ff28959d27 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:24.935429 8100 consensus_queue.cc:260] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:24.935511 8096 tablet_bootstrap.cc:492] T a5e23eebc629494cbb8262bc123a720a P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:24.935794 8100 ts_tablet_manager.cc:1398] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:24.935973 8100 tablet_bootstrap.cc:492] T a5e23eebc629494cbb8262bc123a720a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:24.936940 8096 tablet_bootstrap.cc:654] T a5e23eebc629494cbb8262bc123a720a P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:24.939026 8100 tablet_bootstrap.cc:654] T a5e23eebc629494cbb8262bc123a720a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:24.940245 8096 tablet_bootstrap.cc:492] T a5e23eebc629494cbb8262bc123a720a P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:24.940330 8100 tablet_bootstrap.cc:492] T a5e23eebc629494cbb8262bc123a720a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:24.940353 8096 ts_tablet_manager.cc:1367] T a5e23eebc629494cbb8262bc123a720a P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240503 01:59:24.940405 8100 ts_tablet_manager.cc:1367] T a5e23eebc629494cbb8262bc123a720a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 01:59:24.940836 8100 raft_consensus.cc:348] T a5e23eebc629494cbb8262bc123a720a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:24.940836 8096 raft_consensus.cc:348] T a5e23eebc629494cbb8262bc123a720a P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:24.940966 8100 raft_consensus.cc:374] T a5e23eebc629494cbb8262bc123a720a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:24.940966 8096 raft_consensus.cc:374] T a5e23eebc629494cbb8262bc123a720a P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:24.941001 8100 raft_consensus.cc:729] T a5e23eebc629494cbb8262bc123a720a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:24.941012 8096 raft_consensus.cc:729] T a5e23eebc629494cbb8262bc123a720a P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:24.941130 8096 consensus_queue.cc:260] T a5e23eebc629494cbb8262bc123a720a P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:24.941130 8100 consensus_queue.cc:260] T a5e23eebc629494cbb8262bc123a720a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:24.941542 8100 ts_tablet_manager.cc:1398] T a5e23eebc629494cbb8262bc123a720a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:24.941701 8100 tablet_bootstrap.cc:492] T 4bd8226227c348d5a5d4d8ff28959d27 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:24.942245 8096 ts_tablet_manager.cc:1398] T a5e23eebc629494cbb8262bc123a720a P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 01:59:24.942699 8100 tablet_bootstrap.cc:654] T 4bd8226227c348d5a5d4d8ff28959d27 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:24.943789 8100 tablet_bootstrap.cc:492] T 4bd8226227c348d5a5d4d8ff28959d27 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:24.943872 8100 ts_tablet_manager.cc:1367] T 4bd8226227c348d5a5d4d8ff28959d27 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 01:59:24.944262 8100 raft_consensus.cc:348] T 4bd8226227c348d5a5d4d8ff28959d27 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:24.944401 8100 raft_consensus.cc:374] T 4bd8226227c348d5a5d4d8ff28959d27 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:24.944447 8100 raft_consensus.cc:729] T 4bd8226227c348d5a5d4d8ff28959d27 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:24.944586 8100 consensus_queue.cc:260] T 4bd8226227c348d5a5d4d8ff28959d27 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:24.944833 8100 ts_tablet_manager.cc:1398] T 4bd8226227c348d5a5d4d8ff28959d27 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 01:59:24.981034 8034 raft_consensus.cc:482] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:24.981187 8034 raft_consensus.cc:504] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:24.981647 8034 leader_election.cc:288] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:24.982223 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd8226227c348d5a5d4d8ff28959d27" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:24.982383 7275 raft_consensus.cc:2454] T 4bd8226227c348d5a5d4d8ff28959d27 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:24.982676 6817 leader_election.cc:302] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:24.982879 8034 raft_consensus.cc:2781] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:24.982934 8034 raft_consensus.cc:482] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:24.982961 8034 raft_consensus.cc:3037] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:24.984030 8034 raft_consensus.cc:504] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:24.984433 8034 leader_election.cc:288] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:24.984789 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd8226227c348d5a5d4d8ff28959d27" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:24.984912 7275 raft_consensus.cc:3037] T 4bd8226227c348d5a5d4d8ff28959d27 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:24.985869 7275 raft_consensus.cc:2454] T 4bd8226227c348d5a5d4d8ff28959d27 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:24.986156 6817 leader_election.cc:302] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:24.986167 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd8226227c348d5a5d4d8ff28959d27" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:24.986243 7144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bd8226227c348d5a5d4d8ff28959d27" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:24.986297 7145 raft_consensus.cc:2454] T 4bd8226227c348d5a5d4d8ff28959d27 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:24.986366 8034 raft_consensus.cc:2781] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:24.986460 8034 raft_consensus.cc:686] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 01:59:24.986653 8034 consensus_queue.cc:237] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:24.988337 6738 catalog_manager.cc:5484] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:25.044307 7889 raft_consensus.cc:482] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:25.044481 7889 raft_consensus.cc:504] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:25.044978 7889 leader_election.cc:288] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:25.045382 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ed2793842464443826f83f60fb6dcd1" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:25.045382 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ed2793842464443826f83f60fb6dcd1" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:25.045547 7012 raft_consensus.cc:2454] T 8ed2793842464443826f83f60fb6dcd1 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:25.045547 6881 raft_consensus.cc:2454] T 8ed2793842464443826f83f60fb6dcd1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:25.045833 7078 leader_election.cc:302] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:25.046048 7889 raft_consensus.cc:2781] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:25.046116 7889 raft_consensus.cc:482] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:25.046146 7889 raft_consensus.cc:3037] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:25.047153 7889 raft_consensus.cc:504] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:25.047549 7889 leader_election.cc:288] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Requested vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:25.047941 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ed2793842464443826f83f60fb6dcd1" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:25.047941 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ed2793842464443826f83f60fb6dcd1" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:25.048082 7012 raft_consensus.cc:3037] T 8ed2793842464443826f83f60fb6dcd1 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:25.048082 6881 raft_consensus.cc:3037] T 8ed2793842464443826f83f60fb6dcd1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:25.049255 7012 raft_consensus.cc:2454] T 8ed2793842464443826f83f60fb6dcd1 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:25.049562 6881 raft_consensus.cc:2454] T 8ed2793842464443826f83f60fb6dcd1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:25.049654 7078 leader_election.cc:302] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:25.049849 7889 raft_consensus.cc:2781] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:25.050073 7889 raft_consensus.cc:686] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Becoming Leader. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Running, Role: LEADER
I20240503 01:59:25.050233 7889 consensus_queue.cc:237] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:25.052071 6738 catalog_manager.cc:5484] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:25.158315 8079 raft_consensus.cc:482] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:25.158447 8079 raft_consensus.cc:504] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:25.158946 8079 leader_election.cc:288] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:25.159404 7144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e23eebc629494cbb8262bc123a720a" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:25.159426 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e23eebc629494cbb8262bc123a720a" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:25.159536 7144 raft_consensus.cc:2454] T a5e23eebc629494cbb8262bc123a720a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:25.159569 7012 raft_consensus.cc:2454] T a5e23eebc629494cbb8262bc123a720a P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:25.159845 7211 leader_election.cc:302] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:25.160032 8079 raft_consensus.cc:2781] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:25.160087 8079 raft_consensus.cc:482] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:25.160113 8079 raft_consensus.cc:3037] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:25.160956 8079 raft_consensus.cc:504] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:25.161332 8079 leader_election.cc:288] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Requested vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:25.161697 7144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e23eebc629494cbb8262bc123a720a" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:25.161731 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e23eebc629494cbb8262bc123a720a" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:25.161818 7144 raft_consensus.cc:3037] T a5e23eebc629494cbb8262bc123a720a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:25.161826 7012 raft_consensus.cc:3037] T a5e23eebc629494cbb8262bc123a720a P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:25.162626 7144 raft_consensus.cc:2454] T a5e23eebc629494cbb8262bc123a720a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:25.162962 7211 leader_election.cc:302] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:25.163347 7012 raft_consensus.cc:2454] T a5e23eebc629494cbb8262bc123a720a P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:25.163540 8079 raft_consensus.cc:2781] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:25.163648 8079 raft_consensus.cc:686] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 01:59:25.163900 8079 consensus_queue.cc:237] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:25.166407 6738 catalog_manager.cc:5484] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
W20240503 01:59:25.332715 8115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:25.332998 8115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:25.336290 8115 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 01:59:25.395494 8034 consensus_queue.cc:1035] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:25.406356 7144 raft_consensus.cc:3037] T 4bd8226227c348d5a5d4d8ff28959d27 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:25.407927 7874 consensus_queue.cc:1035] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:25.449098 7012 raft_consensus.cc:1261] T a5e23eebc629494cbb8262bc123a720a P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:25.450615 8080 consensus_queue.cc:1035] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:25.450887 6881 raft_consensus.cc:1261] T 8ed2793842464443826f83f60fb6dcd1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:25.451370 7012 raft_consensus.cc:1261] T 8ed2793842464443826f83f60fb6dcd1 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:25.451905 7889 consensus_queue.cc:1035] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:25.452422 7889 consensus_queue.cc:1035] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:25.457608 7144 raft_consensus.cc:1261] T a5e23eebc629494cbb8262bc123a720a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:25.458122 8080 consensus_queue.cc:1035] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 136.563 ms
time per row: 0.0682817 ms
I20240503 01:59:25.524250 6732 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:25.531560 6993 tablet_service.cc:1467] Processing CreateTablet for tablet 7e0575a7e09741f68340cb92809831f1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=636b9e88e8fd430595fb3edfb428c61b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:25.531764 6863 tablet_service.cc:1467] Processing CreateTablet for tablet 7e0575a7e09741f68340cb92809831f1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=636b9e88e8fd430595fb3edfb428c61b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:25.531955 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e0575a7e09741f68340cb92809831f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:25.532059 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e0575a7e09741f68340cb92809831f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:25.532183 7124 tablet_service.cc:1467] Processing CreateTablet for tablet 7e0575a7e09741f68340cb92809831f1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=636b9e88e8fd430595fb3edfb428c61b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:25.532455 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e0575a7e09741f68340cb92809831f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:25.533797 8178 tablet_bootstrap.cc:492] T 7e0575a7e09741f68340cb92809831f1 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:25.534056 7256 tablet_service.cc:1467] Processing CreateTablet for tablet 6411ac9665194b4f9b9d66f54db0cf25 (DEFAULT_TABLE table=rebalance_test_table_4 [id=636b9e88e8fd430595fb3edfb428c61b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:25.534332 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6411ac9665194b4f9b9d66f54db0cf25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:25.534643 8179 tablet_bootstrap.cc:492] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:25.534823 8178 tablet_bootstrap.cc:654] T 7e0575a7e09741f68340cb92809831f1 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:25.534799 7254 tablet_service.cc:1467] Processing CreateTablet for tablet 24462821d97145ba817a4fefbfe1c710 (DEFAULT_TABLE table=rebalance_test_table_4 [id=636b9e88e8fd430595fb3edfb428c61b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:25.535022 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24462821d97145ba817a4fefbfe1c710. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:25.535627 8179 tablet_bootstrap.cc:654] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:25.537051 8178 tablet_bootstrap.cc:492] T 7e0575a7e09741f68340cb92809831f1 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:25.537106 8179 tablet_bootstrap.cc:492] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:25.537145 8178 ts_tablet_manager.cc:1367] T 7e0575a7e09741f68340cb92809831f1 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:25.537189 8179 ts_tablet_manager.cc:1367] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:25.537164 6863 tablet_service.cc:1467] Processing CreateTablet for tablet 24462821d97145ba817a4fefbfe1c710 (DEFAULT_TABLE table=rebalance_test_table_4 [id=636b9e88e8fd430595fb3edfb428c61b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:25.537425 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24462821d97145ba817a4fefbfe1c710. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:25.537654 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 6411ac9665194b4f9b9d66f54db0cf25 (DEFAULT_TABLE table=rebalance_test_table_4 [id=636b9e88e8fd430595fb3edfb428c61b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:25.537679 8179 raft_consensus.cc:348] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:25.537679 8178 raft_consensus.cc:348] T 7e0575a7e09741f68340cb92809831f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:25.537782 8178 raft_consensus.cc:374] T 7e0575a7e09741f68340cb92809831f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:25.537781 8179 raft_consensus.cc:374] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:25.537813 8178 raft_consensus.cc:729] T 7e0575a7e09741f68340cb92809831f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:25.537817 8179 raft_consensus.cc:729] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:25.537897 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6411ac9665194b4f9b9d66f54db0cf25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:25.537930 8178 consensus_queue.cc:260] T 7e0575a7e09741f68340cb92809831f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:25.537951 8179 consensus_queue.cc:260] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:25.538184 8178 ts_tablet_manager.cc:1398] T 7e0575a7e09741f68340cb92809831f1 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:25.538228 8179 ts_tablet_manager.cc:1398] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:25.539289 8183 tablet_bootstrap.cc:492] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:25.539798 7123 tablet_service.cc:1467] Processing CreateTablet for tablet 6411ac9665194b4f9b9d66f54db0cf25 (DEFAULT_TABLE table=rebalance_test_table_4 [id=636b9e88e8fd430595fb3edfb428c61b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:25.540066 7123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6411ac9665194b4f9b9d66f54db0cf25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:25.540215 6993 tablet_service.cc:1467] Processing CreateTablet for tablet 24462821d97145ba817a4fefbfe1c710 (DEFAULT_TABLE table=rebalance_test_table_4 [id=636b9e88e8fd430595fb3edfb428c61b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:25.540443 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24462821d97145ba817a4fefbfe1c710. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:25.541280 8178 tablet_bootstrap.cc:492] T 24462821d97145ba817a4fefbfe1c710 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:25.542340 8178 tablet_bootstrap.cc:654] T 24462821d97145ba817a4fefbfe1c710 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:25.542698 7968 raft_consensus.cc:482] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:25.542783 7968 raft_consensus.cc:504] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:25.543179 7968 leader_election.cc:288] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:25.543516 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0575a7e09741f68340cb92809831f1" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
W20240503 01:59:25.544075 6949 leader_election.cc:341] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): Illegal state: must be running to vote when last-logged opid is not known
I20240503 01:59:25.544320 8179 tablet_bootstrap.cc:492] T 24462821d97145ba817a4fefbfe1c710 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:25.544526 8180 tablet_bootstrap.cc:492] T 7e0575a7e09741f68340cb92809831f1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:25.544497 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0575a7e09741f68340cb92809831f1" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:25.544610 6881 raft_consensus.cc:2454] T 7e0575a7e09741f68340cb92809831f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:25.544893 8183 tablet_bootstrap.cc:654] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:25.545027 6950 leader_election.cc:302] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a
I20240503 01:59:25.545285 8179 tablet_bootstrap.cc:654] T 24462821d97145ba817a4fefbfe1c710 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:25.546023 8180 tablet_bootstrap.cc:654] T 7e0575a7e09741f68340cb92809831f1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:25.545286 7968 raft_consensus.cc:2781] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:25.546677 7968 raft_consensus.cc:482] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:25.546732 7968 raft_consensus.cc:3037] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:25.547106 8179 tablet_bootstrap.cc:492] T 24462821d97145ba817a4fefbfe1c710 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:25.547201 8179 ts_tablet_manager.cc:1367] T 24462821d97145ba817a4fefbfe1c710 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:25.547233 8178 tablet_bootstrap.cc:492] T 24462821d97145ba817a4fefbfe1c710 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:25.547327 8178 ts_tablet_manager.cc:1367] T 24462821d97145ba817a4fefbfe1c710 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 01:59:25.547659 8179 raft_consensus.cc:348] T 24462821d97145ba817a4fefbfe1c710 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:25.547719 8178 raft_consensus.cc:348] T 24462821d97145ba817a4fefbfe1c710 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:25.547830 8178 raft_consensus.cc:374] T 24462821d97145ba817a4fefbfe1c710 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:25.547797 7968 raft_consensus.cc:504] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:25.547873 8178 raft_consensus.cc:729] T 24462821d97145ba817a4fefbfe1c710 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:25.548019 8178 consensus_queue.cc:260] T 24462821d97145ba817a4fefbfe1c710 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:25.548259 7968 leader_election.cc:288] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:25.548321 8178 ts_tablet_manager.cc:1398] T 24462821d97145ba817a4fefbfe1c710 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:25.548530 8178 tablet_bootstrap.cc:492] T 6411ac9665194b4f9b9d66f54db0cf25 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:25.548842 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0575a7e09741f68340cb92809831f1" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:25.548944 8183 tablet_bootstrap.cc:492] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:25.548955 6881 raft_consensus.cc:3037] T 7e0575a7e09741f68340cb92809831f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:25.549054 8183 ts_tablet_manager.cc:1367] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 01:59:25.549530 8183 raft_consensus.cc:348] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:25.549652 8183 raft_consensus.cc:374] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:25.549686 8183 raft_consensus.cc:729] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:25.549760 8178 tablet_bootstrap.cc:654] T 6411ac9665194b4f9b9d66f54db0cf25 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:25.549865 8183 consensus_queue.cc:260] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:25.549919 6881 raft_consensus.cc:2454] T 7e0575a7e09741f68340cb92809831f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:25.550184 8183 ts_tablet_manager.cc:1398] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:25.550273 6950 leader_election.cc:302] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:25.550364 8183 tablet_bootstrap.cc:492] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:25.550647 7968 raft_consensus.cc:2781] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:25.550700 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0575a7e09741f68340cb92809831f1" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:25.550858 7968 raft_consensus.cc:686] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:25.551017 7968 consensus_queue.cc:237] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
W20240503 01:59:25.551556 6949 leader_election.cc:341] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): Illegal state: must be running to vote when last-logged opid is not known
I20240503 01:59:25.551645 8183 tablet_bootstrap.cc:654] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:25.551826 8180 tablet_bootstrap.cc:492] T 7e0575a7e09741f68340cb92809831f1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:25.552695 8180 ts_tablet_manager.cc:1367] T 7e0575a7e09741f68340cb92809831f1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240503 01:59:25.547785 8179 raft_consensus.cc:374] T 24462821d97145ba817a4fefbfe1c710 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:25.552922 8179 raft_consensus.cc:729] T 24462821d97145ba817a4fefbfe1c710 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:25.553076 8179 consensus_queue.cc:260] T 24462821d97145ba817a4fefbfe1c710 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:25.553381 8179 ts_tablet_manager.cc:1398] T 24462821d97145ba817a4fefbfe1c710 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 01:59:25.553563 8180 raft_consensus.cc:348] T 7e0575a7e09741f68340cb92809831f1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:25.553746 8180 raft_consensus.cc:374] T 7e0575a7e09741f68340cb92809831f1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:25.553828 8180 raft_consensus.cc:729] T 7e0575a7e09741f68340cb92809831f1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:25.554055 8180 consensus_queue.cc:260] T 7e0575a7e09741f68340cb92809831f1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:25.554173 6732 catalog_manager.cc:5484] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:25.554323 8183 tablet_bootstrap.cc:492] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:25.554412 8183 ts_tablet_manager.cc:1367] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:25.554652 8180 ts_tablet_manager.cc:1398] T 7e0575a7e09741f68340cb92809831f1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:25.554858 8180 tablet_bootstrap.cc:492] T 6411ac9665194b4f9b9d66f54db0cf25 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:25.554885 8183 raft_consensus.cc:348] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:25.555006 8183 raft_consensus.cc:374] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:25.555050 8183 raft_consensus.cc:729] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:25.555184 8183 consensus_queue.cc:260] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:25.555457 8183 ts_tablet_manager.cc:1398] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:25.555863 8180 tablet_bootstrap.cc:654] T 6411ac9665194b4f9b9d66f54db0cf25 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:25.557104 8178 tablet_bootstrap.cc:492] T 6411ac9665194b4f9b9d66f54db0cf25 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:25.557211 8178 ts_tablet_manager.cc:1367] T 6411ac9665194b4f9b9d66f54db0cf25 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20240503 01:59:25.557704 8178 raft_consensus.cc:348] T 6411ac9665194b4f9b9d66f54db0cf25 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:25.557825 8178 raft_consensus.cc:374] T 6411ac9665194b4f9b9d66f54db0cf25 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:25.557857 8178 raft_consensus.cc:729] T 6411ac9665194b4f9b9d66f54db0cf25 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:25.557976 8178 consensus_queue.cc:260] T 6411ac9665194b4f9b9d66f54db0cf25 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:25.558276 8178 ts_tablet_manager.cc:1398] T 6411ac9665194b4f9b9d66f54db0cf25 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 01:59:25.558892 8180 tablet_bootstrap.cc:492] T 6411ac9665194b4f9b9d66f54db0cf25 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:25.558986 8180 ts_tablet_manager.cc:1367] T 6411ac9665194b4f9b9d66f54db0cf25 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:25.559463 8180 raft_consensus.cc:348] T 6411ac9665194b4f9b9d66f54db0cf25 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:25.559592 8180 raft_consensus.cc:374] T 6411ac9665194b4f9b9d66f54db0cf25 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:25.559639 8180 raft_consensus.cc:729] T 6411ac9665194b4f9b9d66f54db0cf25 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:25.559769 8180 consensus_queue.cc:260] T 6411ac9665194b4f9b9d66f54db0cf25 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:25.560470 8180 ts_tablet_manager.cc:1398] T 6411ac9665194b4f9b9d66f54db0cf25 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:25.577405 8112 raft_consensus.cc:482] T 7e0575a7e09741f68340cb92809831f1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:25.577555 8112 raft_consensus.cc:504] T 7e0575a7e09741f68340cb92809831f1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:25.577993 8112 leader_election.cc:288] T 7e0575a7e09741f68340cb92809831f1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:25.578481 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0575a7e09741f68340cb92809831f1" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:25.578486 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0575a7e09741f68340cb92809831f1" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:25.578718 6881 raft_consensus.cc:2379] T 7e0575a7e09741f68340cb92809831f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in current term 1: Already voted for candidate 63331f455be049c1bdd6c23c59ac1bef in this term.
I20240503 01:59:25.579469 7081 leader_election.cc:302] T 7e0575a7e09741f68340cb92809831f1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef
I20240503 01:59:25.579684 8112 raft_consensus.cc:3037] T 7e0575a7e09741f68340cb92809831f1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:25.580590 8112 raft_consensus.cc:2726] T 7e0575a7e09741f68340cb92809831f1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 01:59:25.586046 8080 raft_consensus.cc:482] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:25.586179 8080 raft_consensus.cc:504] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:25.586648 8080 leader_election.cc:288] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:25.587196 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24462821d97145ba817a4fefbfe1c710" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:25.587196 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24462821d97145ba817a4fefbfe1c710" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:25.587363 7012 raft_consensus.cc:2454] T 24462821d97145ba817a4fefbfe1c710 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:25.587363 6881 raft_consensus.cc:2454] T 24462821d97145ba817a4fefbfe1c710 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:25.587682 7209 leader_election.cc:302] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:25.587848 8080 raft_consensus.cc:2781] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:25.587924 8080 raft_consensus.cc:482] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:25.587977 8080 raft_consensus.cc:3037] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:25.588805 8080 raft_consensus.cc:504] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:25.589200 8080 leader_election.cc:288] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Requested vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:25.589601 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24462821d97145ba817a4fefbfe1c710" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:25.589743 6881 raft_consensus.cc:3037] T 24462821d97145ba817a4fefbfe1c710 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:25.589732 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24462821d97145ba817a4fefbfe1c710" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:25.589850 7012 raft_consensus.cc:3037] T 24462821d97145ba817a4fefbfe1c710 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:25.590775 7012 raft_consensus.cc:2454] T 24462821d97145ba817a4fefbfe1c710 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:25.590775 6881 raft_consensus.cc:2454] T 24462821d97145ba817a4fefbfe1c710 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:25.591121 7212 leader_election.cc:302] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:25.591291 8080 raft_consensus.cc:2781] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:25.591374 8080 raft_consensus.cc:686] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 01:59:25.591500 8080 consensus_queue.cc:237] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:25.593217 6732 catalog_manager.cc:5484] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:25.649505 8080 raft_consensus.cc:482] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:25.649646 8080 raft_consensus.cc:504] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:25.650210 8080 leader_election.cc:288] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:25.650682 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6411ac9665194b4f9b9d66f54db0cf25" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:25.650683 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6411ac9665194b4f9b9d66f54db0cf25" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:25.650818 6881 raft_consensus.cc:2454] T 6411ac9665194b4f9b9d66f54db0cf25 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:25.650830 7145 raft_consensus.cc:2454] T 6411ac9665194b4f9b9d66f54db0cf25 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:25.651139 7212 leader_election.cc:302] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:25.651314 8080 raft_consensus.cc:2781] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:25.651383 8080 raft_consensus.cc:482] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:25.651424 8080 raft_consensus.cc:3037] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:25.652319 8080 raft_consensus.cc:504] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:25.652693 8080 leader_election.cc:288] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Requested vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:25.653100 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6411ac9665194b4f9b9d66f54db0cf25" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:25.653213 7145 raft_consensus.cc:3037] T 6411ac9665194b4f9b9d66f54db0cf25 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:25.654031 7145 raft_consensus.cc:2454] T 6411ac9665194b4f9b9d66f54db0cf25 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:25.654289 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6411ac9665194b4f9b9d66f54db0cf25" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:25.654445 6881 raft_consensus.cc:3037] T 6411ac9665194b4f9b9d66f54db0cf25 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:25.654497 7211 leader_election.cc:302] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:25.654862 8080 raft_consensus.cc:2781] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:25.655027 8080 raft_consensus.cc:686] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 01:59:25.655361 8080 consensus_queue.cc:237] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:25.655974 6881 raft_consensus.cc:2454] T 6411ac9665194b4f9b9d66f54db0cf25 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:25.657946 6732 catalog_manager.cc:5484] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
W20240503 01:59:25.782405 8195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:25.782851 8195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:25.786949 8195 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 01:59:25.886071 6881 raft_consensus.cc:1261] T 6411ac9665194b4f9b9d66f54db0cf25 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:25.886684 8080 consensus_queue.cc:1035] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:25.887188 7145 raft_consensus.cc:1261] T 7e0575a7e09741f68340cb92809831f1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:25.887257 6881 raft_consensus.cc:1261] T 7e0575a7e09741f68340cb92809831f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:25.887656 8189 consensus_queue.cc:1035] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:25.887734 7145 raft_consensus.cc:1261] T 6411ac9665194b4f9b9d66f54db0cf25 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:25.888285 8189 consensus_queue.cc:1035] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:25.892788 8080 consensus_queue.cc:1035] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:25.907418 7012 raft_consensus.cc:1261] T 24462821d97145ba817a4fefbfe1c710 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:25.907994 8079 consensus_queue.cc:1035] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:25.908572 6882 raft_consensus.cc:1261] T 24462821d97145ba817a4fefbfe1c710 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:25.915771 8079 consensus_queue.cc:1035] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 126.835 ms
time per row: 0.0634174 ms
I20240503 01:59:25.962016 6740 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:25.968297 7124 tablet_service.cc:1467] Processing CreateTablet for tablet d2fbadb911aa4bd7b4db021860c202d3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=eca57da7c1764bd9a55819692641dac8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:25.968664 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2fbadb911aa4bd7b4db021860c202d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:25.968835 6992 tablet_service.cc:1467] Processing CreateTablet for tablet 6d4480c8de204f5397dbdcff91db97f6 (DEFAULT_TABLE table=rebalance_test_table_5 [id=eca57da7c1764bd9a55819692641dac8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:25.969120 6992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d4480c8de204f5397dbdcff91db97f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:25.969533 6994 tablet_service.cc:1467] Processing CreateTablet for tablet 87652406a8e442e58fe636a85ab72158 (DEFAULT_TABLE table=rebalance_test_table_5 [id=eca57da7c1764bd9a55819692641dac8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:25.969764 6994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87652406a8e442e58fe636a85ab72158. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:25.971015 8180 tablet_bootstrap.cc:492] T d2fbadb911aa4bd7b4db021860c202d3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:25.971213 8179 tablet_bootstrap.cc:492] T 6d4480c8de204f5397dbdcff91db97f6 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:25.971751 6993 tablet_service.cc:1467] Processing CreateTablet for tablet d2fbadb911aa4bd7b4db021860c202d3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=eca57da7c1764bd9a55819692641dac8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:25.971995 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2fbadb911aa4bd7b4db021860c202d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:25.971992 8180 tablet_bootstrap.cc:654] T d2fbadb911aa4bd7b4db021860c202d3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:25.972087 8179 tablet_bootstrap.cc:654] T 6d4480c8de204f5397dbdcff91db97f6 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:25.973049 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 87652406a8e442e58fe636a85ab72158 (DEFAULT_TABLE table=rebalance_test_table_5 [id=eca57da7c1764bd9a55819692641dac8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:25.973330 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87652406a8e442e58fe636a85ab72158. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:25.973388 8179 tablet_bootstrap.cc:492] T 6d4480c8de204f5397dbdcff91db97f6 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:25.973480 8179 ts_tablet_manager.cc:1367] T 6d4480c8de204f5397dbdcff91db97f6 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:25.973883 8179 raft_consensus.cc:348] T 6d4480c8de204f5397dbdcff91db97f6 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:25.974010 8179 raft_consensus.cc:374] T 6d4480c8de204f5397dbdcff91db97f6 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:25.974045 8179 raft_consensus.cc:729] T 6d4480c8de204f5397dbdcff91db97f6 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:25.974166 8179 consensus_queue.cc:260] T 6d4480c8de204f5397dbdcff91db97f6 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:25.974429 8179 ts_tablet_manager.cc:1398] T 6d4480c8de204f5397dbdcff91db97f6 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:25.974454 7124 tablet_service.cc:1467] Processing CreateTablet for tablet 6d4480c8de204f5397dbdcff91db97f6 (DEFAULT_TABLE table=rebalance_test_table_5 [id=eca57da7c1764bd9a55819692641dac8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:25.974728 8179 tablet_bootstrap.cc:492] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:25.974735 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d4480c8de204f5397dbdcff91db97f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:25.975500 8178 tablet_bootstrap.cc:492] T 87652406a8e442e58fe636a85ab72158 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:25.975641 8179 tablet_bootstrap.cc:654] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:25.975888 8180 tablet_bootstrap.cc:492] T d2fbadb911aa4bd7b4db021860c202d3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:25.975975 8180 ts_tablet_manager.cc:1367] T d2fbadb911aa4bd7b4db021860c202d3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240503 01:59:25.976374 8178 tablet_bootstrap.cc:654] T 87652406a8e442e58fe636a85ab72158 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:25.976418 8180 raft_consensus.cc:348] T d2fbadb911aa4bd7b4db021860c202d3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:25.976517 8180 raft_consensus.cc:374] T d2fbadb911aa4bd7b4db021860c202d3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:25.976548 8180 raft_consensus.cc:729] T d2fbadb911aa4bd7b4db021860c202d3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:25.976670 8180 consensus_queue.cc:260] T d2fbadb911aa4bd7b4db021860c202d3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:25.976930 8180 ts_tablet_manager.cc:1398] T d2fbadb911aa4bd7b4db021860c202d3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:25.976910 7123 tablet_service.cc:1467] Processing CreateTablet for tablet 87652406a8e442e58fe636a85ab72158 (DEFAULT_TABLE table=rebalance_test_table_5 [id=eca57da7c1764bd9a55819692641dac8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:25.977089 8180 tablet_bootstrap.cc:492] T 6d4480c8de204f5397dbdcff91db97f6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:25.977133 7123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87652406a8e442e58fe636a85ab72158. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:25.977298 8178 tablet_bootstrap.cc:492] T 87652406a8e442e58fe636a85ab72158 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:25.977433 8178 ts_tablet_manager.cc:1367] T 87652406a8e442e58fe636a85ab72158 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 01:59:25.977725 8179 tablet_bootstrap.cc:492] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:25.977818 8179 ts_tablet_manager.cc:1367] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:59:25.977900 8178 raft_consensus.cc:348] T 87652406a8e442e58fe636a85ab72158 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:25.978025 8178 raft_consensus.cc:374] T 87652406a8e442e58fe636a85ab72158 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:25.978070 8178 raft_consensus.cc:729] T 87652406a8e442e58fe636a85ab72158 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:25.978210 8178 consensus_queue.cc:260] T 87652406a8e442e58fe636a85ab72158 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:25.978468 8180 tablet_bootstrap.cc:654] T 6d4480c8de204f5397dbdcff91db97f6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:25.978492 8178 ts_tablet_manager.cc:1398] T 87652406a8e442e58fe636a85ab72158 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:25.978456 8179 raft_consensus.cc:348] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:25.978637 8179 raft_consensus.cc:374] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:25.978673 8179 raft_consensus.cc:729] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:25.978801 8179 consensus_queue.cc:260] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:25.979084 8179 ts_tablet_manager.cc:1398] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 01:59:25.979259 8179 tablet_bootstrap.cc:492] T d2fbadb911aa4bd7b4db021860c202d3 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:25.980103 7256 tablet_service.cc:1467] Processing CreateTablet for tablet 6d4480c8de204f5397dbdcff91db97f6 (DEFAULT_TABLE table=rebalance_test_table_5 [id=eca57da7c1764bd9a55819692641dac8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:25.980202 8179 tablet_bootstrap.cc:654] T d2fbadb911aa4bd7b4db021860c202d3 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:25.980103 7254 tablet_service.cc:1467] Processing CreateTablet for tablet d2fbadb911aa4bd7b4db021860c202d3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=eca57da7c1764bd9a55819692641dac8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:25.980419 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d4480c8de204f5397dbdcff91db97f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:25.981509 8179 tablet_bootstrap.cc:492] T d2fbadb911aa4bd7b4db021860c202d3 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:25.981592 8179 ts_tablet_manager.cc:1367] T d2fbadb911aa4bd7b4db021860c202d3 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:25.982033 8179 raft_consensus.cc:348] T d2fbadb911aa4bd7b4db021860c202d3 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:25.982115 8179 raft_consensus.cc:374] T d2fbadb911aa4bd7b4db021860c202d3 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:25.982134 8179 raft_consensus.cc:729] T d2fbadb911aa4bd7b4db021860c202d3 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:25.982244 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2fbadb911aa4bd7b4db021860c202d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:25.982268 8179 consensus_queue.cc:260] T d2fbadb911aa4bd7b4db021860c202d3 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:25.982615 8179 ts_tablet_manager.cc:1398] T d2fbadb911aa4bd7b4db021860c202d3 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:25.982837 8183 tablet_bootstrap.cc:492] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:25.983546 8183 tablet_bootstrap.cc:654] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:25.984715 8183 tablet_bootstrap.cc:492] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:25.984814 8183 ts_tablet_manager.cc:1367] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:25.985282 8183 raft_consensus.cc:348] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:25.985394 8183 raft_consensus.cc:374] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:25.985425 8183 raft_consensus.cc:729] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:25.985544 8183 consensus_queue.cc:260] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:25.985846 8183 ts_tablet_manager.cc:1398] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:25.986040 8183 tablet_bootstrap.cc:492] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:25.987236 8180 tablet_bootstrap.cc:492] T 6d4480c8de204f5397dbdcff91db97f6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:25.987342 8180 ts_tablet_manager.cc:1367] T 6d4480c8de204f5397dbdcff91db97f6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20240503 01:59:25.987596 8183 tablet_bootstrap.cc:654] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:25.987782 8180 raft_consensus.cc:348] T 6d4480c8de204f5397dbdcff91db97f6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:25.987911 8180 raft_consensus.cc:374] T 6d4480c8de204f5397dbdcff91db97f6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:25.987952 8180 raft_consensus.cc:729] T 6d4480c8de204f5397dbdcff91db97f6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:25.988075 8180 consensus_queue.cc:260] T 6d4480c8de204f5397dbdcff91db97f6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:25.988356 8180 ts_tablet_manager.cc:1398] T 6d4480c8de204f5397dbdcff91db97f6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:25.988533 8180 tablet_bootstrap.cc:492] T 87652406a8e442e58fe636a85ab72158 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:25.988842 8183 tablet_bootstrap.cc:492] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:25.988909 8183 ts_tablet_manager.cc:1367] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:25.989295 8183 raft_consensus.cc:348] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:25.989387 8183 raft_consensus.cc:374] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:25.989409 8183 raft_consensus.cc:729] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:25.989485 8180 tablet_bootstrap.cc:654] T 87652406a8e442e58fe636a85ab72158 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:25.989547 8183 consensus_queue.cc:260] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:25.989812 8183 ts_tablet_manager.cc:1398] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:25.990712 8180 tablet_bootstrap.cc:492] T 87652406a8e442e58fe636a85ab72158 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:25.990803 8180 ts_tablet_manager.cc:1367] T 87652406a8e442e58fe636a85ab72158 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 01:59:25.991266 8180 raft_consensus.cc:348] T 87652406a8e442e58fe636a85ab72158 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:25.991384 8180 raft_consensus.cc:374] T 87652406a8e442e58fe636a85ab72158 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:25.991420 8180 raft_consensus.cc:729] T 87652406a8e442e58fe636a85ab72158 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:25.991564 8180 consensus_queue.cc:260] T 87652406a8e442e58fe636a85ab72158 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:25.991940 8180 ts_tablet_manager.cc:1398] T 87652406a8e442e58fe636a85ab72158 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:26.075753 8189 raft_consensus.cc:482] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:26.075897 8189 raft_consensus.cc:504] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:26.076349 8189 leader_election.cc:288] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:26.076936 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87652406a8e442e58fe636a85ab72158" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:26.077014 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87652406a8e442e58fe636a85ab72158" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:26.077131 7145 raft_consensus.cc:2454] T 87652406a8e442e58fe636a85ab72158 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:26.077139 6881 raft_consensus.cc:2454] T 87652406a8e442e58fe636a85ab72158 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:26.077471 6949 leader_election.cc:302] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:26.077646 8189 raft_consensus.cc:2781] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:26.077710 8189 raft_consensus.cc:482] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:26.077739 8189 raft_consensus.cc:3037] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:26.078573 8189 raft_consensus.cc:504] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:26.078929 8189 leader_election.cc:288] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:26.079358 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87652406a8e442e58fe636a85ab72158" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:26.079357 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87652406a8e442e58fe636a85ab72158" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:26.079494 6881 raft_consensus.cc:3037] T 87652406a8e442e58fe636a85ab72158 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:26.079516 7145 raft_consensus.cc:3037] T 87652406a8e442e58fe636a85ab72158 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:26.080478 6881 raft_consensus.cc:2454] T 87652406a8e442e58fe636a85ab72158 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:26.080478 7145 raft_consensus.cc:2454] T 87652406a8e442e58fe636a85ab72158 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:26.080835 6950 leader_election.cc:302] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:26.081101 8189 raft_consensus.cc:2781] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:26.081194 8189 raft_consensus.cc:686] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:26.081336 8189 consensus_queue.cc:237] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:26.083222 6739 catalog_manager.cc:5484] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:26.157483 8227 raft_consensus.cc:482] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:26.157639 8227 raft_consensus.cc:504] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:26.158058 8227 leader_election.cc:288] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:26.158615 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d4480c8de204f5397dbdcff91db97f6" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:26.158638 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d4480c8de204f5397dbdcff91db97f6" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:26.158759 7145 raft_consensus.cc:2454] T 6d4480c8de204f5397dbdcff91db97f6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:26.158787 7012 raft_consensus.cc:2454] T 6d4480c8de204f5397dbdcff91db97f6 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:26.159066 7211 leader_election.cc:302] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:26.159240 8227 raft_consensus.cc:2781] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:26.159299 8227 raft_consensus.cc:482] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:26.159348 8227 raft_consensus.cc:3037] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:26.160169 8227 raft_consensus.cc:504] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:26.160838 8227 leader_election.cc:288] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Requested vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:26.161381 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d4480c8de204f5397dbdcff91db97f6" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:26.161381 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d4480c8de204f5397dbdcff91db97f6" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:26.161533 7145 raft_consensus.cc:3037] T 6d4480c8de204f5397dbdcff91db97f6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:26.161533 7012 raft_consensus.cc:3037] T 6d4480c8de204f5397dbdcff91db97f6 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:26.162564 7145 raft_consensus.cc:2454] T 6d4480c8de204f5397dbdcff91db97f6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:26.162564 7012 raft_consensus.cc:2454] T 6d4480c8de204f5397dbdcff91db97f6 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:26.162932 7211 leader_election.cc:302] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:26.163152 8227 raft_consensus.cc:2781] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:26.163249 8227 raft_consensus.cc:686] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 01:59:26.163408 8227 consensus_queue.cc:237] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:26.165194 6739 catalog_manager.cc:5484] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:26.225356 8227 raft_consensus.cc:482] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:26.225513 8227 raft_consensus.cc:504] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:26.225929 8227 leader_election.cc:288] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:26.226438 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2fbadb911aa4bd7b4db021860c202d3" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:26.226450 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2fbadb911aa4bd7b4db021860c202d3" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:26.226614 7012 raft_consensus.cc:2454] T d2fbadb911aa4bd7b4db021860c202d3 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:26.226614 7145 raft_consensus.cc:2454] T d2fbadb911aa4bd7b4db021860c202d3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:26.226924 7211 leader_election.cc:302] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:26.227109 8227 raft_consensus.cc:2781] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:26.227169 8227 raft_consensus.cc:482] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:26.227209 8227 raft_consensus.cc:3037] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:26.228013 8227 raft_consensus.cc:504] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:26.228343 8227 leader_election.cc:288] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Requested vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:26.228716 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2fbadb911aa4bd7b4db021860c202d3" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:26.228734 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2fbadb911aa4bd7b4db021860c202d3" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:26.228832 7012 raft_consensus.cc:3037] T d2fbadb911aa4bd7b4db021860c202d3 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:26.228853 7145 raft_consensus.cc:3037] T d2fbadb911aa4bd7b4db021860c202d3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:26.229630 7012 raft_consensus.cc:2454] T d2fbadb911aa4bd7b4db021860c202d3 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:26.229672 7145 raft_consensus.cc:2454] T d2fbadb911aa4bd7b4db021860c202d3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:26.229915 7209 leader_election.cc:302] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:26.230103 8228 raft_consensus.cc:2781] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:26.230198 8228 raft_consensus.cc:686] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 01:59:26.230329 8228 consensus_queue.cc:237] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:26.232033 6739 catalog_manager.cc:5484] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
W20240503 01:59:26.389776 8243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:26.390108 8243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:26.393457 8243 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 01:59:26.478350 7145 raft_consensus.cc:1261] T 6d4480c8de204f5397dbdcff91db97f6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:26.478457 7014 raft_consensus.cc:1261] T 6d4480c8de204f5397dbdcff91db97f6 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:26.479699 8228 consensus_queue.cc:1035] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:26.481277 8227 consensus_queue.cc:1035] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:26.483914 7012 raft_consensus.cc:1261] T d2fbadb911aa4bd7b4db021860c202d3 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:26.484769 7144 raft_consensus.cc:1261] T d2fbadb911aa4bd7b4db021860c202d3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:26.485498 8227 consensus_queue.cc:1035] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:26.485843 8227 consensus_queue.cc:1035] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:26.488209 8189 consensus_queue.cc:1035] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:26.492014 7143 raft_consensus.cc:1261] T 87652406a8e442e58fe636a85ab72158 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:26.493952 8189 consensus_queue.cc:1035] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 106.297 ms
time per row: 0.0531486 ms
I20240503 01:59:26.554458 6736 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:26.560684 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 7343f90f35f1419ab7e1da2fc80f6881 (DEFAULT_TABLE table=rebalance_test_table_6 [id=d4b0f87e5f7a49e897a3904f2d2e871c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:26.561040 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7343f90f35f1419ab7e1da2fc80f6881. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:26.560976 6993 tablet_service.cc:1467] Processing CreateTablet for tablet 7343f90f35f1419ab7e1da2fc80f6881 (DEFAULT_TABLE table=rebalance_test_table_6 [id=d4b0f87e5f7a49e897a3904f2d2e871c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:26.561261 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7343f90f35f1419ab7e1da2fc80f6881. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:26.561256 7254 tablet_service.cc:1467] Processing CreateTablet for tablet 7343f90f35f1419ab7e1da2fc80f6881 (DEFAULT_TABLE table=rebalance_test_table_6 [id=d4b0f87e5f7a49e897a3904f2d2e871c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:26.561543 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7343f90f35f1419ab7e1da2fc80f6881. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:26.561724 6863 tablet_service.cc:1467] Processing CreateTablet for tablet e24b3cae166f459888fb8b2aef96f942 (DEFAULT_TABLE table=rebalance_test_table_6 [id=d4b0f87e5f7a49e897a3904f2d2e871c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:26.561954 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e24b3cae166f459888fb8b2aef96f942. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:26.562798 6862 tablet_service.cc:1467] Processing CreateTablet for tablet 5ddc47f7144b4cfdbd9a496b5c7562e8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=d4b0f87e5f7a49e897a3904f2d2e871c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:26.563036 6862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ddc47f7144b4cfdbd9a496b5c7562e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:26.563577 8303 tablet_bootstrap.cc:492] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:26.563779 8304 tablet_bootstrap.cc:492] T 7343f90f35f1419ab7e1da2fc80f6881 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:26.563802 7123 tablet_service.cc:1467] Processing CreateTablet for tablet e24b3cae166f459888fb8b2aef96f942 (DEFAULT_TABLE table=rebalance_test_table_6 [id=d4b0f87e5f7a49e897a3904f2d2e871c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:26.564055 7123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e24b3cae166f459888fb8b2aef96f942. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:26.564589 6993 tablet_service.cc:1467] Processing CreateTablet for tablet 5ddc47f7144b4cfdbd9a496b5c7562e8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=d4b0f87e5f7a49e897a3904f2d2e871c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:26.564687 8304 tablet_bootstrap.cc:654] T 7343f90f35f1419ab7e1da2fc80f6881 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:26.564831 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ddc47f7144b4cfdbd9a496b5c7562e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:26.566063 8303 tablet_bootstrap.cc:654] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:26.566903 7256 tablet_service.cc:1467] Processing CreateTablet for tablet 5ddc47f7144b4cfdbd9a496b5c7562e8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=d4b0f87e5f7a49e897a3904f2d2e871c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:26.567044 7254 tablet_service.cc:1467] Processing CreateTablet for tablet e24b3cae166f459888fb8b2aef96f942 (DEFAULT_TABLE table=rebalance_test_table_6 [id=d4b0f87e5f7a49e897a3904f2d2e871c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:26.567157 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ddc47f7144b4cfdbd9a496b5c7562e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:26.567167 8303 tablet_bootstrap.cc:492] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:26.567281 8303 ts_tablet_manager.cc:1367] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 01:59:26.567332 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e24b3cae166f459888fb8b2aef96f942. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:26.567737 8303 raft_consensus.cc:348] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:26.567853 8303 raft_consensus.cc:374] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:26.567883 8303 raft_consensus.cc:729] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:26.568004 8303 consensus_queue.cc:260] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:26.568311 8303 ts_tablet_manager.cc:1398] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:26.569052 8303 tablet_bootstrap.cc:492] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:26.569622 8306 tablet_bootstrap.cc:492] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:26.569991 8303 tablet_bootstrap.cc:654] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:26.570642 8306 tablet_bootstrap.cc:654] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:26.571192 8303 tablet_bootstrap.cc:492] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:26.571311 8303 ts_tablet_manager.cc:1367] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:26.571852 8306 tablet_bootstrap.cc:492] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:26.571852 8304 tablet_bootstrap.cc:492] T 7343f90f35f1419ab7e1da2fc80f6881 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:26.571949 8306 ts_tablet_manager.cc:1367] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:26.571949 8304 ts_tablet_manager.cc:1367] T 7343f90f35f1419ab7e1da2fc80f6881 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 01:59:26.572398 8306 raft_consensus.cc:348] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:26.572398 8304 raft_consensus.cc:348] T 7343f90f35f1419ab7e1da2fc80f6881 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:26.572520 8304 raft_consensus.cc:374] T 7343f90f35f1419ab7e1da2fc80f6881 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:26.572520 8306 raft_consensus.cc:374] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:26.572567 8304 raft_consensus.cc:729] T 7343f90f35f1419ab7e1da2fc80f6881 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:26.572572 8306 raft_consensus.cc:729] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:26.572705 8306 consensus_queue.cc:260] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:26.572705 8304 consensus_queue.cc:260] T 7343f90f35f1419ab7e1da2fc80f6881 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:26.572980 8306 ts_tablet_manager.cc:1398] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:26.573053 8304 ts_tablet_manager.cc:1398] T 7343f90f35f1419ab7e1da2fc80f6881 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:26.573433 8303 raft_consensus.cc:348] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:26.573551 8303 raft_consensus.cc:374] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:26.573585 8303 raft_consensus.cc:729] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:26.573783 8304 tablet_bootstrap.cc:492] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:26.573740 8303 consensus_queue.cc:260] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:26.574110 8303 ts_tablet_manager.cc:1398] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 01:59:26.574290 8303 tablet_bootstrap.cc:492] T e24b3cae166f459888fb8b2aef96f942 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:26.574776 8304 tablet_bootstrap.cc:654] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:26.575732 8303 tablet_bootstrap.cc:654] T e24b3cae166f459888fb8b2aef96f942 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:26.575959 8304 tablet_bootstrap.cc:492] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:26.576045 8304 ts_tablet_manager.cc:1367] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:26.576604 8304 raft_consensus.cc:348] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:26.576730 8304 raft_consensus.cc:374] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:26.576764 8304 raft_consensus.cc:729] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:26.576814 8303 tablet_bootstrap.cc:492] T e24b3cae166f459888fb8b2aef96f942 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:26.576880 8303 ts_tablet_manager.cc:1367] T e24b3cae166f459888fb8b2aef96f942 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:26.576906 8304 consensus_queue.cc:260] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:26.577188 8304 ts_tablet_manager.cc:1398] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:26.577304 8303 raft_consensus.cc:348] T e24b3cae166f459888fb8b2aef96f942 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:26.577442 8303 raft_consensus.cc:374] T e24b3cae166f459888fb8b2aef96f942 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:26.577482 8303 raft_consensus.cc:729] T e24b3cae166f459888fb8b2aef96f942 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:26.577620 8303 consensus_queue.cc:260] T e24b3cae166f459888fb8b2aef96f942 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:26.577873 8303 ts_tablet_manager.cc:1398] T e24b3cae166f459888fb8b2aef96f942 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:26.588819 8312 tablet_bootstrap.cc:492] T 7343f90f35f1419ab7e1da2fc80f6881 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:26.589646 8312 tablet_bootstrap.cc:654] T 7343f90f35f1419ab7e1da2fc80f6881 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:26.590620 8312 tablet_bootstrap.cc:492] T 7343f90f35f1419ab7e1da2fc80f6881 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:26.590713 8312 ts_tablet_manager.cc:1367] T 7343f90f35f1419ab7e1da2fc80f6881 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:26.591096 8312 raft_consensus.cc:348] T 7343f90f35f1419ab7e1da2fc80f6881 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:26.591213 8312 raft_consensus.cc:374] T 7343f90f35f1419ab7e1da2fc80f6881 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:26.591249 8312 raft_consensus.cc:729] T 7343f90f35f1419ab7e1da2fc80f6881 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:26.591393 8312 consensus_queue.cc:260] T 7343f90f35f1419ab7e1da2fc80f6881 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:26.591642 8312 ts_tablet_manager.cc:1398] T 7343f90f35f1419ab7e1da2fc80f6881 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:26.591806 8312 tablet_bootstrap.cc:492] T e24b3cae166f459888fb8b2aef96f942 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:26.592423 8301 raft_consensus.cc:482] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:26.592542 8301 raft_consensus.cc:504] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:26.592628 8312 tablet_bootstrap.cc:654] T e24b3cae166f459888fb8b2aef96f942 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:26.592984 8301 leader_election.cc:288] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:26.593294 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e24b3cae166f459888fb8b2aef96f942" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:26.593418 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e24b3cae166f459888fb8b2aef96f942" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:26.593537 7275 raft_consensus.cc:2454] T e24b3cae166f459888fb8b2aef96f942 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:26.593924 8312 tablet_bootstrap.cc:492] T e24b3cae166f459888fb8b2aef96f942 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:26.594033 8312 ts_tablet_manager.cc:1367] T e24b3cae166f459888fb8b2aef96f942 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:26.594228 7079 leader_election.cc:302] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:26.594422 8301 raft_consensus.cc:2781] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:26.594491 8301 raft_consensus.cc:482] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:26.594559 8301 raft_consensus.cc:3037] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:26.594506 8312 raft_consensus.cc:348] T e24b3cae166f459888fb8b2aef96f942 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:26.594691 8312 raft_consensus.cc:374] T e24b3cae166f459888fb8b2aef96f942 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:26.594734 8312 raft_consensus.cc:729] T e24b3cae166f459888fb8b2aef96f942 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:26.594846 8312 consensus_queue.cc:260] T e24b3cae166f459888fb8b2aef96f942 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:26.595139 8312 ts_tablet_manager.cc:1398] T e24b3cae166f459888fb8b2aef96f942 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:26.595306 8312 tablet_bootstrap.cc:492] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:26.595535 8301 raft_consensus.cc:504] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:26.595888 8301 leader_election.cc:288] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Requested vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:26.596263 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e24b3cae166f459888fb8b2aef96f942" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:26.596333 8312 tablet_bootstrap.cc:654] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:26.596397 7275 raft_consensus.cc:3037] T e24b3cae166f459888fb8b2aef96f942 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:26.597427 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e24b3cae166f459888fb8b2aef96f942" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:26.597471 7275 raft_consensus.cc:2454] T e24b3cae166f459888fb8b2aef96f942 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:26.597544 6881 raft_consensus.cc:3037] T e24b3cae166f459888fb8b2aef96f942 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:26.598083 7079 leader_election.cc:302] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:26.598451 8301 raft_consensus.cc:2781] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:26.598569 8301 raft_consensus.cc:686] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Becoming Leader. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Running, Role: LEADER
I20240503 01:59:26.598585 6881 raft_consensus.cc:2454] T e24b3cae166f459888fb8b2aef96f942 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:26.598721 8301 consensus_queue.cc:237] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
W20240503 01:59:26.599089 7081 leader_election.cc:341] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): Illegal state: must be running to vote when last-logged opid is not known
I20240503 01:59:26.600184 8312 tablet_bootstrap.cc:492] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:26.600288 8312 ts_tablet_manager.cc:1367] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:26.600541 6737 catalog_manager.cc:5484] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:26.600787 8312 raft_consensus.cc:348] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:26.600893 8312 raft_consensus.cc:374] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:26.600934 8312 raft_consensus.cc:729] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:26.601078 8312 consensus_queue.cc:260] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:26.601411 8312 ts_tablet_manager.cc:1398] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:26.694161 8189 raft_consensus.cc:482] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:26.694315 8189 raft_consensus.cc:504] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:26.694859 8189 leader_election.cc:288] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:26.695484 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddc47f7144b4cfdbd9a496b5c7562e8" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:26.695713 6881 raft_consensus.cc:2454] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:26.696149 6950 leader_election.cc:302] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:26.696367 8189 raft_consensus.cc:2781] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:26.696426 8189 raft_consensus.cc:482] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:26.696470 8189 raft_consensus.cc:3037] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:26.697321 8189 raft_consensus.cc:504] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:26.697746 8189 leader_election.cc:288] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:26.698239 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddc47f7144b4cfdbd9a496b5c7562e8" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:26.698396 6881 raft_consensus.cc:3037] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:26.699221 6881 raft_consensus.cc:2454] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:26.699242 7276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddc47f7144b4cfdbd9a496b5c7562e8" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:26.699242 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ddc47f7144b4cfdbd9a496b5c7562e8" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:26.699411 7276 raft_consensus.cc:3037] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:26.699542 6950 leader_election.cc:302] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:26.699769 8189 raft_consensus.cc:2781] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:26.699865 8189 raft_consensus.cc:686] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:26.700013 8189 consensus_queue.cc:237] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:26.700374 7276 raft_consensus.cc:2454] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:26.701813 6736 catalog_manager.cc:5484] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:26.807157 8079 raft_consensus.cc:482] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:26.807323 8079 raft_consensus.cc:504] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:26.807787 8079 leader_election.cc:288] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:26.808506 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7343f90f35f1419ab7e1da2fc80f6881" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:26.808475 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7343f90f35f1419ab7e1da2fc80f6881" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:26.808746 6881 raft_consensus.cc:2454] T 7343f90f35f1419ab7e1da2fc80f6881 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:26.808759 7012 raft_consensus.cc:2454] T 7343f90f35f1419ab7e1da2fc80f6881 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:26.809098 7212 leader_election.cc:302] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:26.809401 8079 raft_consensus.cc:2781] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:26.809490 8079 raft_consensus.cc:482] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:26.809535 8079 raft_consensus.cc:3037] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:26.810504 8079 raft_consensus.cc:504] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:26.810926 8079 leader_election.cc:288] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Requested vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:26.811410 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7343f90f35f1419ab7e1da2fc80f6881" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:26.811458 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7343f90f35f1419ab7e1da2fc80f6881" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:26.811559 7012 raft_consensus.cc:3037] T 7343f90f35f1419ab7e1da2fc80f6881 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:26.811601 6881 raft_consensus.cc:3037] T 7343f90f35f1419ab7e1da2fc80f6881 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:26.812366 7012 raft_consensus.cc:2454] T 7343f90f35f1419ab7e1da2fc80f6881 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:26.812386 6881 raft_consensus.cc:2454] T 7343f90f35f1419ab7e1da2fc80f6881 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:26.812724 7212 leader_election.cc:302] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:26.812911 8079 raft_consensus.cc:2781] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:26.813161 8079 raft_consensus.cc:686] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 01:59:26.813387 8079 consensus_queue.cc:237] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:26.815811 6736 catalog_manager.cc:5484] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
W20240503 01:59:26.977998 8320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:26.978309 8320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:26.981717 8320 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 01:59:27.074681 6881 raft_consensus.cc:1261] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:27.074681 7276 raft_consensus.cc:1261] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:27.075136 8189 consensus_queue.cc:1035] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:27.075330 7968 consensus_queue.cc:1035] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:27.077730 7012 raft_consensus.cc:1261] T 7343f90f35f1419ab7e1da2fc80f6881 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:27.078347 8228 consensus_queue.cc:1035] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:27.081498 6881 raft_consensus.cc:1261] T 7343f90f35f1419ab7e1da2fc80f6881 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:27.082646 8228 consensus_queue.cc:1035] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:27.090095 6881 raft_consensus.cc:1261] T e24b3cae166f459888fb8b2aef96f942 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:27.090224 7275 raft_consensus.cc:1261] T e24b3cae166f459888fb8b2aef96f942 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:27.090508 8301 consensus_queue.cc:1035] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:27.093688 8112 consensus_queue.cc:1035] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:27.108443 8366 mvcc.cc:205] Tried to move back new op lower bound from 7023417618793836544 to 7023417616789889024. Current Snapshot: MvccSnapshot[applied={T|T < 7023417618793836544}]
I20240503 01:59:27.108981 8355 mvcc.cc:205] Tried to move back new op lower bound from 7023417618808676352 to 7023417617204494336. Current Snapshot: MvccSnapshot[applied={T|T < 7023417618808676352}]
INSERT report
rows total: 2000
time total: 121.823 ms
time per row: 0.0609114 ms
I20240503 01:59:27.156868 6736 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:27.163337 7123 tablet_service.cc:1467] Processing CreateTablet for tablet 2065dcd6cde84f9a928e4b817737dd42 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3428bac98e0c49908450b3b51cb9d18c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:27.163729 7123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2065dcd6cde84f9a928e4b817737dd42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:27.163816 6993 tablet_service.cc:1467] Processing CreateTablet for tablet 2065dcd6cde84f9a928e4b817737dd42 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3428bac98e0c49908450b3b51cb9d18c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:27.163844 6863 tablet_service.cc:1467] Processing CreateTablet for tablet 2065dcd6cde84f9a928e4b817737dd42 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3428bac98e0c49908450b3b51cb9d18c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:27.164098 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2065dcd6cde84f9a928e4b817737dd42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:27.164283 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2065dcd6cde84f9a928e4b817737dd42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:27.164518 6861 tablet_service.cc:1467] Processing CreateTablet for tablet c029253a832e46a48ae16691fc044502 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3428bac98e0c49908450b3b51cb9d18c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:27.164901 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c029253a832e46a48ae16691fc044502. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:27.166944 8384 tablet_bootstrap.cc:492] T 2065dcd6cde84f9a928e4b817737dd42 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:27.167357 7254 tablet_service.cc:1467] Processing CreateTablet for tablet 76e8697ac8624e03a46bddae35f4c9fa (DEFAULT_TABLE table=rebalance_test_table_7 [id=3428bac98e0c49908450b3b51cb9d18c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:27.167482 7123 tablet_service.cc:1467] Processing CreateTablet for tablet 76e8697ac8624e03a46bddae35f4c9fa (DEFAULT_TABLE table=rebalance_test_table_7 [id=3428bac98e0c49908450b3b51cb9d18c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:27.167640 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76e8697ac8624e03a46bddae35f4c9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:27.167708 7123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76e8697ac8624e03a46bddae35f4c9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:27.167889 8384 tablet_bootstrap.cc:654] T 2065dcd6cde84f9a928e4b817737dd42 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:27.169359 8386 tablet_bootstrap.cc:492] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:27.170150 7256 tablet_service.cc:1467] Processing CreateTablet for tablet c029253a832e46a48ae16691fc044502 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3428bac98e0c49908450b3b51cb9d18c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:27.170392 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c029253a832e46a48ae16691fc044502. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:27.170447 8386 tablet_bootstrap.cc:654] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:27.171216 8385 tablet_bootstrap.cc:492] T 2065dcd6cde84f9a928e4b817737dd42 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:27.170207 7124 tablet_service.cc:1467] Processing CreateTablet for tablet c029253a832e46a48ae16691fc044502 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3428bac98e0c49908450b3b51cb9d18c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:27.171620 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c029253a832e46a48ae16691fc044502. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:27.172281 8385 tablet_bootstrap.cc:654] T 2065dcd6cde84f9a928e4b817737dd42 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:27.173288 6993 tablet_service.cc:1467] Processing CreateTablet for tablet 76e8697ac8624e03a46bddae35f4c9fa (DEFAULT_TABLE table=rebalance_test_table_7 [id=3428bac98e0c49908450b3b51cb9d18c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:27.173506 8386 tablet_bootstrap.cc:492] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:27.173569 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76e8697ac8624e03a46bddae35f4c9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:27.173595 8386 ts_tablet_manager.cc:1367] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:27.174033 8386 raft_consensus.cc:348] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:27.174140 8386 raft_consensus.cc:374] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:27.174175 8386 raft_consensus.cc:729] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:27.174309 8386 consensus_queue.cc:260] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:27.174644 8386 ts_tablet_manager.cc:1398] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:27.175149 8384 tablet_bootstrap.cc:492] T 2065dcd6cde84f9a928e4b817737dd42 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:27.175241 8384 ts_tablet_manager.cc:1367] T 2065dcd6cde84f9a928e4b817737dd42 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 01:59:27.175251 8386 tablet_bootstrap.cc:492] T c029253a832e46a48ae16691fc044502 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:27.175722 8384 raft_consensus.cc:348] T 2065dcd6cde84f9a928e4b817737dd42 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:27.175830 8384 raft_consensus.cc:374] T 2065dcd6cde84f9a928e4b817737dd42 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:27.175869 8384 raft_consensus.cc:729] T 2065dcd6cde84f9a928e4b817737dd42 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:27.176009 8384 consensus_queue.cc:260] T 2065dcd6cde84f9a928e4b817737dd42 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:27.176318 8384 ts_tablet_manager.cc:1398] T 2065dcd6cde84f9a928e4b817737dd42 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:27.176465 8384 tablet_bootstrap.cc:492] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:27.176789 8386 tablet_bootstrap.cc:654] T c029253a832e46a48ae16691fc044502 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:27.177410 8384 tablet_bootstrap.cc:654] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:27.178418 8388 tablet_bootstrap.cc:492] T 76e8697ac8624e03a46bddae35f4c9fa P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:27.178961 8384 tablet_bootstrap.cc:492] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:27.179104 8384 ts_tablet_manager.cc:1367] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:27.179165 8385 tablet_bootstrap.cc:492] T 2065dcd6cde84f9a928e4b817737dd42 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:27.179246 8386 tablet_bootstrap.cc:492] T c029253a832e46a48ae16691fc044502 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:27.179275 8385 ts_tablet_manager.cc:1367] T 2065dcd6cde84f9a928e4b817737dd42 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 01:59:27.179332 8386 ts_tablet_manager.cc:1367] T c029253a832e46a48ae16691fc044502 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:27.179438 8388 tablet_bootstrap.cc:654] T 76e8697ac8624e03a46bddae35f4c9fa P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:27.179772 8386 raft_consensus.cc:348] T c029253a832e46a48ae16691fc044502 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:27.179772 8385 raft_consensus.cc:348] T 2065dcd6cde84f9a928e4b817737dd42 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:27.179915 8385 raft_consensus.cc:374] T 2065dcd6cde84f9a928e4b817737dd42 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:27.179915 8386 raft_consensus.cc:374] T c029253a832e46a48ae16691fc044502 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:27.179958 8386 raft_consensus.cc:729] T c029253a832e46a48ae16691fc044502 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:27.179958 8385 raft_consensus.cc:729] T 2065dcd6cde84f9a928e4b817737dd42 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:27.180097 8385 consensus_queue.cc:260] T 2065dcd6cde84f9a928e4b817737dd42 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:27.180140 8386 consensus_queue.cc:260] T c029253a832e46a48ae16691fc044502 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:27.180387 8385 ts_tablet_manager.cc:1398] T 2065dcd6cde84f9a928e4b817737dd42 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:27.179850 8384 raft_consensus.cc:348] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:27.180539 8385 tablet_bootstrap.cc:492] T 76e8697ac8624e03a46bddae35f4c9fa P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:27.180611 8384 raft_consensus.cc:374] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:27.180699 8384 raft_consensus.cc:729] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:27.180960 8384 consensus_queue.cc:260] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:27.181484 8385 tablet_bootstrap.cc:654] T 76e8697ac8624e03a46bddae35f4c9fa P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:27.181883 8384 ts_tablet_manager.cc:1398] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:27.182060 8384 tablet_bootstrap.cc:492] T c029253a832e46a48ae16691fc044502 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:27.183144 8384 tablet_bootstrap.cc:654] T c029253a832e46a48ae16691fc044502 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:27.183224 8385 tablet_bootstrap.cc:492] T 76e8697ac8624e03a46bddae35f4c9fa P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:27.183303 8385 ts_tablet_manager.cc:1367] T 76e8697ac8624e03a46bddae35f4c9fa P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:27.183737 8386 ts_tablet_manager.cc:1398] T c029253a832e46a48ae16691fc044502 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:27.183795 8385 raft_consensus.cc:348] T 76e8697ac8624e03a46bddae35f4c9fa P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:27.183907 8385 raft_consensus.cc:374] T 76e8697ac8624e03a46bddae35f4c9fa P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:27.183938 8385 raft_consensus.cc:729] T 76e8697ac8624e03a46bddae35f4c9fa P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:27.184077 8385 consensus_queue.cc:260] T 76e8697ac8624e03a46bddae35f4c9fa P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:27.184445 8385 ts_tablet_manager.cc:1398] T 76e8697ac8624e03a46bddae35f4c9fa P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:27.185451 8384 tablet_bootstrap.cc:492] T c029253a832e46a48ae16691fc044502 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:27.185552 8384 ts_tablet_manager.cc:1367] T c029253a832e46a48ae16691fc044502 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:27.186029 8384 raft_consensus.cc:348] T c029253a832e46a48ae16691fc044502 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:27.186141 8384 raft_consensus.cc:374] T c029253a832e46a48ae16691fc044502 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:27.186174 8384 raft_consensus.cc:729] T c029253a832e46a48ae16691fc044502 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:27.186336 8384 consensus_queue.cc:260] T c029253a832e46a48ae16691fc044502 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:27.187182 8388 tablet_bootstrap.cc:492] T 76e8697ac8624e03a46bddae35f4c9fa P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:27.187309 8388 ts_tablet_manager.cc:1367] T 76e8697ac8624e03a46bddae35f4c9fa P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 01:59:27.187898 8388 raft_consensus.cc:348] T 76e8697ac8624e03a46bddae35f4c9fa P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:27.188024 8388 raft_consensus.cc:374] T 76e8697ac8624e03a46bddae35f4c9fa P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:27.188056 8388 raft_consensus.cc:729] T 76e8697ac8624e03a46bddae35f4c9fa P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:27.188180 8388 consensus_queue.cc:260] T 76e8697ac8624e03a46bddae35f4c9fa P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:27.188482 8388 ts_tablet_manager.cc:1398] T 76e8697ac8624e03a46bddae35f4c9fa P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:27.188482 8384 ts_tablet_manager.cc:1398] T c029253a832e46a48ae16691fc044502 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:27.188665 8388 tablet_bootstrap.cc:492] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:27.189916 8388 tablet_bootstrap.cc:654] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:27.191221 8388 tablet_bootstrap.cc:492] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:27.191327 8388 ts_tablet_manager.cc:1367] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:27.192325 8388 raft_consensus.cc:348] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:27.192487 8388 raft_consensus.cc:374] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:27.192526 8388 raft_consensus.cc:729] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:27.193037 8388 consensus_queue.cc:260] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:27.193379 8388 ts_tablet_manager.cc:1398] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:27.225947 8174 raft_consensus.cc:482] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:27.226133 8174 raft_consensus.cc:504] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:27.226668 8174 leader_election.cc:288] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:27.227129 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2065dcd6cde84f9a928e4b817737dd42" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:27.227133 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2065dcd6cde84f9a928e4b817737dd42" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:27.227304 7012 raft_consensus.cc:2454] T 2065dcd6cde84f9a928e4b817737dd42 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:27.227306 7143 raft_consensus.cc:2454] T 2065dcd6cde84f9a928e4b817737dd42 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:27.227574 6816 leader_election.cc:302] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:27.227751 8174 raft_consensus.cc:2781] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:27.227804 8174 raft_consensus.cc:482] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:27.227830 8174 raft_consensus.cc:3037] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:27.228699 8174 raft_consensus.cc:504] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:27.229099 8174 leader_election.cc:288] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Requested vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:27.229609 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2065dcd6cde84f9a928e4b817737dd42" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:27.229725 7143 raft_consensus.cc:3037] T 2065dcd6cde84f9a928e4b817737dd42 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:27.229630 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2065dcd6cde84f9a928e4b817737dd42" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:27.229840 7012 raft_consensus.cc:3037] T 2065dcd6cde84f9a928e4b817737dd42 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:27.230875 7143 raft_consensus.cc:2454] T 2065dcd6cde84f9a928e4b817737dd42 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:27.230988 7012 raft_consensus.cc:2454] T 2065dcd6cde84f9a928e4b817737dd42 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:27.231401 6816 leader_election.cc:302] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:27.231595 8174 raft_consensus.cc:2781] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:27.231830 8174 raft_consensus.cc:686] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 01:59:27.232237 8174 consensus_queue.cc:237] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:27.234234 6732 catalog_manager.cc:5484] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:27.255293 8360 raft_consensus.cc:482] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:27.255446 8360 raft_consensus.cc:504] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:27.255977 8360 leader_election.cc:288] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:27.256255 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c029253a832e46a48ae16691fc044502" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:27.256397 7143 raft_consensus.cc:2454] T c029253a832e46a48ae16691fc044502 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:27.256390 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c029253a832e46a48ae16691fc044502" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:27.256578 6882 raft_consensus.cc:2454] T c029253a832e46a48ae16691fc044502 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:27.256705 7211 leader_election.cc:302] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:27.256884 8360 raft_consensus.cc:2781] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:27.256933 8360 raft_consensus.cc:482] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:27.256984 8360 raft_consensus.cc:3037] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:27.257970 8360 raft_consensus.cc:504] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:27.258284 8360 leader_election.cc:288] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Requested vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:27.258666 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c029253a832e46a48ae16691fc044502" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:27.258806 6882 raft_consensus.cc:3037] T c029253a832e46a48ae16691fc044502 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:27.258944 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c029253a832e46a48ae16691fc044502" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:27.259117 7143 raft_consensus.cc:3037] T c029253a832e46a48ae16691fc044502 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:27.259696 6882 raft_consensus.cc:2454] T c029253a832e46a48ae16691fc044502 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:27.259932 7143 raft_consensus.cc:2454] T c029253a832e46a48ae16691fc044502 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:27.259994 7212 leader_election.cc:302] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:27.260192 8360 raft_consensus.cc:2781] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:27.260291 8360 raft_consensus.cc:686] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 01:59:27.260452 8360 consensus_queue.cc:237] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:27.262791 6732 catalog_manager.cc:5484] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:27.307305 8112 raft_consensus.cc:482] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:27.307444 8112 raft_consensus.cc:504] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:27.307880 8112 leader_election.cc:288] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:27.308423 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76e8697ac8624e03a46bddae35f4c9fa" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:27.308568 7275 raft_consensus.cc:2454] T 76e8697ac8624e03a46bddae35f4c9fa P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:27.308570 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76e8697ac8624e03a46bddae35f4c9fa" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:27.308746 7012 raft_consensus.cc:2454] T 76e8697ac8624e03a46bddae35f4c9fa P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:27.308907 7079 leader_election.cc:302] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:27.309108 8112 raft_consensus.cc:2781] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:27.309181 8112 raft_consensus.cc:482] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:27.309232 8112 raft_consensus.cc:3037] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:27.310140 8112 raft_consensus.cc:504] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:27.310604 8112 leader_election.cc:288] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Requested vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:27.311097 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76e8697ac8624e03a46bddae35f4c9fa" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:27.311229 7012 raft_consensus.cc:3037] T 76e8697ac8624e03a46bddae35f4c9fa P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:27.311298 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76e8697ac8624e03a46bddae35f4c9fa" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:27.311435 7275 raft_consensus.cc:3037] T 76e8697ac8624e03a46bddae35f4c9fa P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:27.312131 7012 raft_consensus.cc:2454] T 76e8697ac8624e03a46bddae35f4c9fa P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:27.312201 7275 raft_consensus.cc:2454] T 76e8697ac8624e03a46bddae35f4c9fa P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:27.312476 7078 leader_election.cc:302] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:27.312927 8112 raft_consensus.cc:2781] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:27.313019 8112 raft_consensus.cc:686] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Becoming Leader. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Running, Role: LEADER
I20240503 01:59:27.313303 8112 consensus_queue.cc:237] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:27.315393 6732 catalog_manager.cc:5484] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
W20240503 01:59:27.446020 8401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:27.446309 8401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:27.449621 8401 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 01:59:27.564208 7014 raft_consensus.cc:1261] T 76e8697ac8624e03a46bddae35f4c9fa P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:27.564677 7275 raft_consensus.cc:1261] T 76e8697ac8624e03a46bddae35f4c9fa P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:27.566185 8301 consensus_queue.cc:1035] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:27.566936 7014 raft_consensus.cc:1261] T 2065dcd6cde84f9a928e4b817737dd42 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:27.567152 7143 raft_consensus.cc:1261] T 2065dcd6cde84f9a928e4b817737dd42 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:27.567416 8397 consensus_queue.cc:1035] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:27.567588 8112 consensus_queue.cc:1035] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:27.568200 8397 consensus_queue.cc:1035] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:27.578155 6882 raft_consensus.cc:1261] T c029253a832e46a48ae16691fc044502 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:27.581223 8354 mvcc.cc:205] Tried to move back new op lower bound from 7023417620738879488 to 7023417619716874240. Current Snapshot: MvccSnapshot[applied={T|T < 7023417620738879488 or (T in {7023417620738879488})}]
I20240503 01:59:27.583212 7145 raft_consensus.cc:1261] T c029253a832e46a48ae16691fc044502 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:27.592451 8228 consensus_queue.cc:1035] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:27.593183 8228 consensus_queue.cc:1035] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:27.600507 8356 mvcc.cc:205] Tried to move back new op lower bound from 7023417620738879488 to 7023417619716874240. Current Snapshot: MvccSnapshot[applied={T|T < 7023417620738879488}]
INSERT report
rows total: 2000
time total: 157.495 ms
time per row: 0.0787475 ms
I20240503 01:59:27.666312 6738 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:27.676182 7124 tablet_service.cc:1467] Processing CreateTablet for tablet 98be5e3f127842d1b6380f13cb6842dd (DEFAULT_TABLE table=rebalance_test_table_8 [id=0f89521df2e14bcd8bbc6f1041196932]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:27.676523 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98be5e3f127842d1b6380f13cb6842dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:27.677618 6993 tablet_service.cc:1467] Processing CreateTablet for tablet 98be5e3f127842d1b6380f13cb6842dd (DEFAULT_TABLE table=rebalance_test_table_8 [id=0f89521df2e14bcd8bbc6f1041196932]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:27.677963 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98be5e3f127842d1b6380f13cb6842dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:27.679210 6994 tablet_service.cc:1467] Processing CreateTablet for tablet 46f4a187af854867bcd509c17ced5cd7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=0f89521df2e14bcd8bbc6f1041196932]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:27.679179 7256 tablet_service.cc:1467] Processing CreateTablet for tablet 46f4a187af854867bcd509c17ced5cd7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=0f89521df2e14bcd8bbc6f1041196932]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:27.679443 6994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46f4a187af854867bcd509c17ced5cd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:27.679448 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46f4a187af854867bcd509c17ced5cd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:27.681563 8384 tablet_bootstrap.cc:492] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:27.681587 7254 tablet_service.cc:1467] Processing CreateTablet for tablet 836882aa1fac4721be3cf096540eedc4 (DEFAULT_TABLE table=rebalance_test_table_8 [id=0f89521df2e14bcd8bbc6f1041196932]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:27.681808 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 836882aa1fac4721be3cf096540eedc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:27.681742 6993 tablet_service.cc:1467] Processing CreateTablet for tablet 836882aa1fac4721be3cf096540eedc4 (DEFAULT_TABLE table=rebalance_test_table_8 [id=0f89521df2e14bcd8bbc6f1041196932]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:27.682029 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 836882aa1fac4721be3cf096540eedc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:27.682559 8384 tablet_bootstrap.cc:654] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:27.682806 8385 tablet_bootstrap.cc:492] T 98be5e3f127842d1b6380f13cb6842dd P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:27.683775 8385 tablet_bootstrap.cc:654] T 98be5e3f127842d1b6380f13cb6842dd P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:27.684414 8388 tablet_bootstrap.cc:492] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:27.684435 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 98be5e3f127842d1b6380f13cb6842dd (DEFAULT_TABLE table=rebalance_test_table_8 [id=0f89521df2e14bcd8bbc6f1041196932]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:27.684688 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98be5e3f127842d1b6380f13cb6842dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:27.685253 8388 tablet_bootstrap.cc:654] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:27.686452 8388 tablet_bootstrap.cc:492] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:27.686594 8388 ts_tablet_manager.cc:1367] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:27.686926 6863 tablet_service.cc:1467] Processing CreateTablet for tablet 46f4a187af854867bcd509c17ced5cd7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=0f89521df2e14bcd8bbc6f1041196932]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:27.687163 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46f4a187af854867bcd509c17ced5cd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:27.687153 8388 raft_consensus.cc:348] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:27.687285 8388 raft_consensus.cc:374] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:27.687325 8388 raft_consensus.cc:729] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:27.687477 8388 consensus_queue.cc:260] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:27.687973 8388 ts_tablet_manager.cc:1398] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:27.688201 8388 tablet_bootstrap.cc:492] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:27.689122 8386 tablet_bootstrap.cc:492] T 46f4a187af854867bcd509c17ced5cd7 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:27.689270 8388 tablet_bootstrap.cc:654] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:27.689857 8386 tablet_bootstrap.cc:654] T 46f4a187af854867bcd509c17ced5cd7 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:27.690611 8388 tablet_bootstrap.cc:492] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:27.690716 8388 ts_tablet_manager.cc:1367] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:27.691339 8388 raft_consensus.cc:348] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:27.691475 8388 raft_consensus.cc:374] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:27.691514 8388 raft_consensus.cc:729] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:27.691665 8388 consensus_queue.cc:260] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:27.692059 8388 ts_tablet_manager.cc:1398] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:27.693830 7124 tablet_service.cc:1467] Processing CreateTablet for tablet 836882aa1fac4721be3cf096540eedc4 (DEFAULT_TABLE table=rebalance_test_table_8 [id=0f89521df2e14bcd8bbc6f1041196932]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:27.694154 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 836882aa1fac4721be3cf096540eedc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:27.696969 8385 tablet_bootstrap.cc:492] T 98be5e3f127842d1b6380f13cb6842dd P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:27.697088 8385 ts_tablet_manager.cc:1367] T 98be5e3f127842d1b6380f13cb6842dd P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 01:59:27.697217 8384 tablet_bootstrap.cc:492] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:27.697316 8384 ts_tablet_manager.cc:1367] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 01:59:27.697330 8386 tablet_bootstrap.cc:492] T 46f4a187af854867bcd509c17ced5cd7 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:27.697439 8386 ts_tablet_manager.cc:1367] T 46f4a187af854867bcd509c17ced5cd7 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 01:59:27.697669 8385 raft_consensus.cc:348] T 98be5e3f127842d1b6380f13cb6842dd P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:27.697788 8385 raft_consensus.cc:374] T 98be5e3f127842d1b6380f13cb6842dd P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:27.697820 8385 raft_consensus.cc:729] T 98be5e3f127842d1b6380f13cb6842dd P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:27.697912 8386 raft_consensus.cc:348] T 46f4a187af854867bcd509c17ced5cd7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:27.697943 8385 consensus_queue.cc:260] T 98be5e3f127842d1b6380f13cb6842dd P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:27.698004 8386 raft_consensus.cc:374] T 46f4a187af854867bcd509c17ced5cd7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:27.697919 8384 raft_consensus.cc:348] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:27.698035 8386 raft_consensus.cc:729] T 46f4a187af854867bcd509c17ced5cd7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:27.698061 8384 raft_consensus.cc:374] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:27.698093 8384 raft_consensus.cc:729] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:27.698153 8386 consensus_queue.cc:260] T 46f4a187af854867bcd509c17ced5cd7 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:27.698249 8385 ts_tablet_manager.cc:1398] T 98be5e3f127842d1b6380f13cb6842dd P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:27.698216 8384 consensus_queue.cc:260] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:27.698407 8386 ts_tablet_manager.cc:1398] T 46f4a187af854867bcd509c17ced5cd7 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:27.698437 8385 tablet_bootstrap.cc:492] T 46f4a187af854867bcd509c17ced5cd7 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:27.698477 8384 ts_tablet_manager.cc:1398] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:27.698590 8386 tablet_bootstrap.cc:492] T 98be5e3f127842d1b6380f13cb6842dd P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:27.699362 8384 tablet_bootstrap.cc:492] T 836882aa1fac4721be3cf096540eedc4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:27.699786 8386 tablet_bootstrap.cc:654] T 98be5e3f127842d1b6380f13cb6842dd P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:27.700645 8384 tablet_bootstrap.cc:654] T 836882aa1fac4721be3cf096540eedc4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:27.700982 8385 tablet_bootstrap.cc:654] T 46f4a187af854867bcd509c17ced5cd7 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:27.702754 8384 tablet_bootstrap.cc:492] T 836882aa1fac4721be3cf096540eedc4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:27.702889 8384 ts_tablet_manager.cc:1367] T 836882aa1fac4721be3cf096540eedc4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:27.703487 8384 raft_consensus.cc:348] T 836882aa1fac4721be3cf096540eedc4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:27.703639 8384 raft_consensus.cc:374] T 836882aa1fac4721be3cf096540eedc4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:27.703684 8384 raft_consensus.cc:729] T 836882aa1fac4721be3cf096540eedc4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:27.703831 8384 consensus_queue.cc:260] T 836882aa1fac4721be3cf096540eedc4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:27.704258 8384 ts_tablet_manager.cc:1398] T 836882aa1fac4721be3cf096540eedc4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:27.706395 8386 tablet_bootstrap.cc:492] T 98be5e3f127842d1b6380f13cb6842dd P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:27.706534 8386 ts_tablet_manager.cc:1367] T 98be5e3f127842d1b6380f13cb6842dd P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 01:59:27.707111 8386 raft_consensus.cc:348] T 98be5e3f127842d1b6380f13cb6842dd P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:27.707235 8386 raft_consensus.cc:374] T 98be5e3f127842d1b6380f13cb6842dd P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:27.707268 8386 raft_consensus.cc:729] T 98be5e3f127842d1b6380f13cb6842dd P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:27.707406 8386 consensus_queue.cc:260] T 98be5e3f127842d1b6380f13cb6842dd P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:27.707494 8385 tablet_bootstrap.cc:492] T 46f4a187af854867bcd509c17ced5cd7 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:27.707599 8385 ts_tablet_manager.cc:1367] T 46f4a187af854867bcd509c17ced5cd7 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 01:59:27.707712 8386 ts_tablet_manager.cc:1398] T 98be5e3f127842d1b6380f13cb6842dd P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:27.708072 8385 raft_consensus.cc:348] T 46f4a187af854867bcd509c17ced5cd7 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:27.708171 8385 raft_consensus.cc:374] T 46f4a187af854867bcd509c17ced5cd7 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:27.708202 8385 raft_consensus.cc:729] T 46f4a187af854867bcd509c17ced5cd7 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:27.708583 8385 consensus_queue.cc:260] T 46f4a187af854867bcd509c17ced5cd7 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:27.708978 8385 ts_tablet_manager.cc:1398] T 46f4a187af854867bcd509c17ced5cd7 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:27.709189 8385 tablet_bootstrap.cc:492] T 836882aa1fac4721be3cf096540eedc4 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:27.711166 8385 tablet_bootstrap.cc:654] T 836882aa1fac4721be3cf096540eedc4 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:27.715281 8385 tablet_bootstrap.cc:492] T 836882aa1fac4721be3cf096540eedc4 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:27.715404 8385 ts_tablet_manager.cc:1367] T 836882aa1fac4721be3cf096540eedc4 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:27.715814 8385 raft_consensus.cc:348] T 836882aa1fac4721be3cf096540eedc4 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:27.715921 8385 raft_consensus.cc:374] T 836882aa1fac4721be3cf096540eedc4 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:27.715957 8385 raft_consensus.cc:729] T 836882aa1fac4721be3cf096540eedc4 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:27.716151 8385 consensus_queue.cc:260] T 836882aa1fac4721be3cf096540eedc4 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:27.716603 8385 ts_tablet_manager.cc:1398] T 836882aa1fac4721be3cf096540eedc4 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:27.722551 8228 raft_consensus.cc:482] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:27.722719 8228 raft_consensus.cc:504] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:27.723206 8228 leader_election.cc:288] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:27.723708 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "836882aa1fac4721be3cf096540eedc4" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:27.723860 7014 raft_consensus.cc:2454] T 836882aa1fac4721be3cf096540eedc4 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:27.724220 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "836882aa1fac4721be3cf096540eedc4" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:27.724328 7209 leader_election.cc:302] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:27.724364 7143 raft_consensus.cc:2454] T 836882aa1fac4721be3cf096540eedc4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:27.725183 8228 raft_consensus.cc:2781] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:27.725245 8228 raft_consensus.cc:482] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:27.725272 8228 raft_consensus.cc:3037] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:27.726183 8228 raft_consensus.cc:504] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:27.726564 8228 leader_election.cc:288] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Requested vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:27.726905 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "836882aa1fac4721be3cf096540eedc4" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:27.727034 7014 raft_consensus.cc:3037] T 836882aa1fac4721be3cf096540eedc4 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:27.727919 7014 raft_consensus.cc:2454] T 836882aa1fac4721be3cf096540eedc4 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:27.728220 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "836882aa1fac4721be3cf096540eedc4" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:27.728351 7143 raft_consensus.cc:3037] T 836882aa1fac4721be3cf096540eedc4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:27.728431 7209 leader_election.cc:302] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:27.728783 8228 raft_consensus.cc:2781] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:27.729070 8228 raft_consensus.cc:686] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 01:59:27.729221 8228 consensus_queue.cc:237] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:27.729380 7143 raft_consensus.cc:2454] T 836882aa1fac4721be3cf096540eedc4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:27.732481 6740 catalog_manager.cc:5484] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:27.973618 8301 raft_consensus.cc:482] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:27.973814 8301 raft_consensus.cc:504] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:27.974340 8301 leader_election.cc:288] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:27.974750 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98be5e3f127842d1b6380f13cb6842dd" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:27.974828 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98be5e3f127842d1b6380f13cb6842dd" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:27.974902 6882 raft_consensus.cc:2454] T 98be5e3f127842d1b6380f13cb6842dd P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:27.974957 7014 raft_consensus.cc:2454] T 98be5e3f127842d1b6380f13cb6842dd P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:27.975240 7081 leader_election.cc:302] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:27.975412 8301 raft_consensus.cc:2781] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:27.975498 8301 raft_consensus.cc:482] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:27.975554 8301 raft_consensus.cc:3037] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:27.976433 8301 raft_consensus.cc:504] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:27.976797 8301 leader_election.cc:288] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Requested vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:27.977236 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98be5e3f127842d1b6380f13cb6842dd" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:27.977391 6882 raft_consensus.cc:3037] T 98be5e3f127842d1b6380f13cb6842dd P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:27.977357 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98be5e3f127842d1b6380f13cb6842dd" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:27.977597 7014 raft_consensus.cc:3037] T 98be5e3f127842d1b6380f13cb6842dd P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:27.978313 6882 raft_consensus.cc:2454] T 98be5e3f127842d1b6380f13cb6842dd P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:27.978611 7014 raft_consensus.cc:2454] T 98be5e3f127842d1b6380f13cb6842dd P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:27.978709 7081 leader_election.cc:302] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:27.978981 8301 raft_consensus.cc:2781] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:27.979127 8301 raft_consensus.cc:686] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Becoming Leader. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Running, Role: LEADER
I20240503 01:59:27.979418 8301 consensus_queue.cc:237] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:27.981534 6740 catalog_manager.cc:5484] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:28.003208 8228 raft_consensus.cc:482] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:28.003350 8228 raft_consensus.cc:504] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:28.003803 8228 leader_election.cc:288] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:28.004419 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f4a187af854867bcd509c17ced5cd7" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:28.004495 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f4a187af854867bcd509c17ced5cd7" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:28.004598 7014 raft_consensus.cc:2454] T 46f4a187af854867bcd509c17ced5cd7 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:28.004731 6882 raft_consensus.cc:2454] T 46f4a187af854867bcd509c17ced5cd7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:28.004992 7209 leader_election.cc:302] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:28.005156 8228 raft_consensus.cc:2781] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:28.005230 8228 raft_consensus.cc:482] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:28.005268 8228 raft_consensus.cc:3037] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:28.006211 8228 raft_consensus.cc:504] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:28.006666 8228 leader_election.cc:288] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Requested vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:28.007112 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f4a187af854867bcd509c17ced5cd7" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:28.007109 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f4a187af854867bcd509c17ced5cd7" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:28.007251 6882 raft_consensus.cc:3037] T 46f4a187af854867bcd509c17ced5cd7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:28.007251 7014 raft_consensus.cc:3037] T 46f4a187af854867bcd509c17ced5cd7 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:28.008230 6882 raft_consensus.cc:2454] T 46f4a187af854867bcd509c17ced5cd7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:28.008230 7014 raft_consensus.cc:2454] T 46f4a187af854867bcd509c17ced5cd7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:28.008589 7212 leader_election.cc:302] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:28.008736 8079 raft_consensus.cc:2781] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:28.008824 8079 raft_consensus.cc:686] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 01:59:28.008965 8079 consensus_queue.cc:237] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:28.010941 6740 catalog_manager.cc:5484] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:28.183267 8360 consensus_queue.cc:1035] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:28.187060 8228 consensus_queue.cc:1035] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 01:59:28.205602 8451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:28.205904 8451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:28.209249 8451 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 01:59:28.303272 7012 raft_consensus.cc:1261] T 98be5e3f127842d1b6380f13cb6842dd P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:28.303426 6882 raft_consensus.cc:1261] T 98be5e3f127842d1b6380f13cb6842dd P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:28.304004 8112 consensus_queue.cc:1035] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:28.305006 8301 consensus_queue.cc:1035] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:28.308043 6881 raft_consensus.cc:1261] T 46f4a187af854867bcd509c17ced5cd7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:28.308615 8079 consensus_queue.cc:1035] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:28.310457 7013 raft_consensus.cc:1261] T 46f4a187af854867bcd509c17ced5cd7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:28.314905 8228 consensus_queue.cc:1035] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 120.238 ms
time per row: 0.0601188 ms
I20240503 01:59:28.381362 6740 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:28.388813 7124 tablet_service.cc:1467] Processing CreateTablet for tablet 4600418bf64c4a0d962831c6924c46e1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=f050ea79f4334dcda01dce5d3e804176]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:28.388813 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 4600418bf64c4a0d962831c6924c46e1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=f050ea79f4334dcda01dce5d3e804176]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:28.389158 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4600418bf64c4a0d962831c6924c46e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:28.389158 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4600418bf64c4a0d962831c6924c46e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:28.389140 7254 tablet_service.cc:1467] Processing CreateTablet for tablet 4600418bf64c4a0d962831c6924c46e1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=f050ea79f4334dcda01dce5d3e804176]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:28.389482 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4600418bf64c4a0d962831c6924c46e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:28.389621 7256 tablet_service.cc:1467] Processing CreateTablet for tablet 72e090a6c18e4a13bbb343eb40a07694 (DEFAULT_TABLE table=rebalance_test_table_9 [id=f050ea79f4334dcda01dce5d3e804176]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:28.389835 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e090a6c18e4a13bbb343eb40a07694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:28.391252 7125 tablet_service.cc:1467] Processing CreateTablet for tablet 5cb5072e91534671ad3b4dd85f93fa5b (DEFAULT_TABLE table=rebalance_test_table_9 [id=f050ea79f4334dcda01dce5d3e804176]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:28.391502 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cb5072e91534671ad3b4dd85f93fa5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:28.391819 7123 tablet_service.cc:1467] Processing CreateTablet for tablet 72e090a6c18e4a13bbb343eb40a07694 (DEFAULT_TABLE table=rebalance_test_table_9 [id=f050ea79f4334dcda01dce5d3e804176]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:28.391899 8510 tablet_bootstrap.cc:492] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:28.392028 7123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e090a6c18e4a13bbb343eb40a07694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:28.392834 8510 tablet_bootstrap.cc:654] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:28.393710 8511 tablet_bootstrap.cc:492] T 4600418bf64c4a0d962831c6924c46e1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:28.394340 8510 tablet_bootstrap.cc:492] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:28.394451 8510 ts_tablet_manager.cc:1367] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:28.394778 6863 tablet_service.cc:1467] Processing CreateTablet for tablet 5cb5072e91534671ad3b4dd85f93fa5b (DEFAULT_TABLE table=rebalance_test_table_9 [id=f050ea79f4334dcda01dce5d3e804176]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:28.394975 8510 raft_consensus.cc:348] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:28.395074 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cb5072e91534671ad3b4dd85f93fa5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:28.395109 8510 raft_consensus.cc:374] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:28.395146 8510 raft_consensus.cc:729] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:28.395274 8510 consensus_queue.cc:260] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:28.395354 8511 tablet_bootstrap.cc:654] T 4600418bf64c4a0d962831c6924c46e1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:28.395568 8510 ts_tablet_manager.cc:1398] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:28.395751 8510 tablet_bootstrap.cc:492] T 72e090a6c18e4a13bbb343eb40a07694 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:28.395696 6993 tablet_service.cc:1467] Processing CreateTablet for tablet 5cb5072e91534671ad3b4dd85f93fa5b (DEFAULT_TABLE table=rebalance_test_table_9 [id=f050ea79f4334dcda01dce5d3e804176]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:28.395988 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cb5072e91534671ad3b4dd85f93fa5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:28.396514 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 72e090a6c18e4a13bbb343eb40a07694 (DEFAULT_TABLE table=rebalance_test_table_9 [id=f050ea79f4334dcda01dce5d3e804176]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:28.396795 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e090a6c18e4a13bbb343eb40a07694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:28.396968 8510 tablet_bootstrap.cc:654] T 72e090a6c18e4a13bbb343eb40a07694 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:28.394766 8509 tablet_bootstrap.cc:492] T 4600418bf64c4a0d962831c6924c46e1 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:28.399937 8510 tablet_bootstrap.cc:492] T 72e090a6c18e4a13bbb343eb40a07694 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:28.400035 8510 ts_tablet_manager.cc:1367] T 72e090a6c18e4a13bbb343eb40a07694 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:28.400440 8509 tablet_bootstrap.cc:654] T 4600418bf64c4a0d962831c6924c46e1 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:28.400454 8510 raft_consensus.cc:348] T 72e090a6c18e4a13bbb343eb40a07694 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:28.400557 8511 tablet_bootstrap.cc:492] T 4600418bf64c4a0d962831c6924c46e1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:28.400573 8510 raft_consensus.cc:374] T 72e090a6c18e4a13bbb343eb40a07694 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:28.400609 8510 raft_consensus.cc:729] T 72e090a6c18e4a13bbb343eb40a07694 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:28.400668 8511 ts_tablet_manager.cc:1367] T 4600418bf64c4a0d962831c6924c46e1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:59:28.400759 8510 consensus_queue.cc:260] T 72e090a6c18e4a13bbb343eb40a07694 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:28.401073 8510 ts_tablet_manager.cc:1398] T 72e090a6c18e4a13bbb343eb40a07694 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:28.401221 8511 raft_consensus.cc:348] T 4600418bf64c4a0d962831c6924c46e1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:28.401319 8511 raft_consensus.cc:374] T 4600418bf64c4a0d962831c6924c46e1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:28.401341 8511 raft_consensus.cc:729] T 4600418bf64c4a0d962831c6924c46e1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:28.401500 8511 consensus_queue.cc:260] T 4600418bf64c4a0d962831c6924c46e1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:28.401583 8515 tablet_bootstrap.cc:492] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:28.402328 8511 ts_tablet_manager.cc:1398] T 4600418bf64c4a0d962831c6924c46e1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:28.402705 8511 tablet_bootstrap.cc:492] T 5cb5072e91534671ad3b4dd85f93fa5b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:28.402709 8515 tablet_bootstrap.cc:654] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:28.403187 8509 tablet_bootstrap.cc:492] T 4600418bf64c4a0d962831c6924c46e1 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:28.403283 8509 ts_tablet_manager.cc:1367] T 4600418bf64c4a0d962831c6924c46e1 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 01:59:28.403769 8509 raft_consensus.cc:348] T 4600418bf64c4a0d962831c6924c46e1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:28.403914 8509 raft_consensus.cc:374] T 4600418bf64c4a0d962831c6924c46e1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:28.403936 8511 tablet_bootstrap.cc:654] T 5cb5072e91534671ad3b4dd85f93fa5b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:28.403949 8509 raft_consensus.cc:729] T 4600418bf64c4a0d962831c6924c46e1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:28.404099 8509 consensus_queue.cc:260] T 4600418bf64c4a0d962831c6924c46e1 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:28.404412 8509 ts_tablet_manager.cc:1398] T 4600418bf64c4a0d962831c6924c46e1 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:28.404588 8509 tablet_bootstrap.cc:492] T 5cb5072e91534671ad3b4dd85f93fa5b P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:28.405135 8511 tablet_bootstrap.cc:492] T 5cb5072e91534671ad3b4dd85f93fa5b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:28.405228 8511 ts_tablet_manager.cc:1367] T 5cb5072e91534671ad3b4dd85f93fa5b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:28.405831 8511 raft_consensus.cc:348] T 5cb5072e91534671ad3b4dd85f93fa5b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:28.406114 8509 tablet_bootstrap.cc:654] T 5cb5072e91534671ad3b4dd85f93fa5b P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:28.406117 8511 raft_consensus.cc:374] T 5cb5072e91534671ad3b4dd85f93fa5b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:28.406181 8511 raft_consensus.cc:729] T 5cb5072e91534671ad3b4dd85f93fa5b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:28.406857 8515 tablet_bootstrap.cc:492] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:28.406968 8515 ts_tablet_manager.cc:1367] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 01:59:28.406352 8511 consensus_queue.cc:260] T 5cb5072e91534671ad3b4dd85f93fa5b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:28.407526 8515 raft_consensus.cc:348] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:28.407655 8515 raft_consensus.cc:374] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:28.407693 8515 raft_consensus.cc:729] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:28.407714 8511 ts_tablet_manager.cc:1398] T 5cb5072e91534671ad3b4dd85f93fa5b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:28.407824 8515 consensus_queue.cc:260] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:28.407889 8511 tablet_bootstrap.cc:492] T 72e090a6c18e4a13bbb343eb40a07694 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:28.408128 8515 ts_tablet_manager.cc:1398] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:28.408788 8511 tablet_bootstrap.cc:654] T 72e090a6c18e4a13bbb343eb40a07694 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:28.410156 8511 tablet_bootstrap.cc:492] T 72e090a6c18e4a13bbb343eb40a07694 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:28.410166 8509 tablet_bootstrap.cc:492] T 5cb5072e91534671ad3b4dd85f93fa5b P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:28.410248 8511 ts_tablet_manager.cc:1367] T 72e090a6c18e4a13bbb343eb40a07694 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:28.410249 8509 ts_tablet_manager.cc:1367] T 5cb5072e91534671ad3b4dd85f93fa5b P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:28.410774 8509 raft_consensus.cc:348] T 5cb5072e91534671ad3b4dd85f93fa5b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:28.410758 8511 raft_consensus.cc:348] T 72e090a6c18e4a13bbb343eb40a07694 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:28.410871 8511 raft_consensus.cc:374] T 72e090a6c18e4a13bbb343eb40a07694 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:28.410871 8509 raft_consensus.cc:374] T 5cb5072e91534671ad3b4dd85f93fa5b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:28.410903 8511 raft_consensus.cc:729] T 72e090a6c18e4a13bbb343eb40a07694 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:28.410903 8509 raft_consensus.cc:729] T 5cb5072e91534671ad3b4dd85f93fa5b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:28.411046 8511 consensus_queue.cc:260] T 72e090a6c18e4a13bbb343eb40a07694 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:28.411082 8509 consensus_queue.cc:260] T 5cb5072e91534671ad3b4dd85f93fa5b P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:28.411370 8511 ts_tablet_manager.cc:1398] T 72e090a6c18e4a13bbb343eb40a07694 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:28.411437 8509 ts_tablet_manager.cc:1398] T 5cb5072e91534671ad3b4dd85f93fa5b P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:28.411593 8509 tablet_bootstrap.cc:492] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:28.412657 8509 tablet_bootstrap.cc:654] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:28.414176 8509 tablet_bootstrap.cc:492] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:28.414269 8509 ts_tablet_manager.cc:1367] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:28.414795 8509 raft_consensus.cc:348] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:28.414903 8509 raft_consensus.cc:374] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:28.414937 8509 raft_consensus.cc:729] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:28.415067 8509 consensus_queue.cc:260] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:28.415307 8509 ts_tablet_manager.cc:1398] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:28.415658 8364 raft_consensus.cc:482] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:28.415774 8364 raft_consensus.cc:504] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:28.416167 8364 leader_election.cc:288] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:28.416535 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cb5072e91534671ad3b4dd85f93fa5b" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:28.416538 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cb5072e91534671ad3b4dd85f93fa5b" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:28.416646 7145 raft_consensus.cc:2454] T 5cb5072e91534671ad3b4dd85f93fa5b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:28.416646 6881 raft_consensus.cc:2454] T 5cb5072e91534671ad3b4dd85f93fa5b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:28.416955 6950 leader_election.cc:302] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:28.417114 8364 raft_consensus.cc:2781] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:28.417165 8364 raft_consensus.cc:482] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:28.417224 8364 raft_consensus.cc:3037] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:28.418238 8364 raft_consensus.cc:504] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:28.418637 8364 leader_election.cc:288] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:28.418898 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cb5072e91534671ad3b4dd85f93fa5b" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:28.419045 6881 raft_consensus.cc:3037] T 5cb5072e91534671ad3b4dd85f93fa5b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:28.419798 6881 raft_consensus.cc:2454] T 5cb5072e91534671ad3b4dd85f93fa5b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:28.420035 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cb5072e91534671ad3b4dd85f93fa5b" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:28.420096 6950 leader_election.cc:302] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:28.420162 7145 raft_consensus.cc:3037] T 5cb5072e91534671ad3b4dd85f93fa5b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:28.420289 8364 raft_consensus.cc:2781] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:28.420529 8364 raft_consensus.cc:686] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:28.420681 8364 consensus_queue.cc:237] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:28.421043 7145 raft_consensus.cc:2454] T 5cb5072e91534671ad3b4dd85f93fa5b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:28.423008 6740 catalog_manager.cc:5484] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:28.432564 8397 raft_consensus.cc:482] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:28.432730 8397 raft_consensus.cc:504] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:28.433136 8397 leader_election.cc:288] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:28.433578 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e090a6c18e4a13bbb343eb40a07694" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:28.433579 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e090a6c18e4a13bbb343eb40a07694" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:28.433725 7145 raft_consensus.cc:2454] T 72e090a6c18e4a13bbb343eb40a07694 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:28.433715 7275 raft_consensus.cc:2454] T 72e090a6c18e4a13bbb343eb40a07694 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:28.434401 6818 leader_election.cc:302] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:28.434624 8397 raft_consensus.cc:2781] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:28.434695 8397 raft_consensus.cc:482] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:28.434729 8397 raft_consensus.cc:3037] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:28.435770 8397 raft_consensus.cc:504] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:28.436182 8397 leader_election.cc:288] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Requested vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:28.436489 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e090a6c18e4a13bbb343eb40a07694" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:28.436564 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e090a6c18e4a13bbb343eb40a07694" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:28.436623 7275 raft_consensus.cc:3037] T 72e090a6c18e4a13bbb343eb40a07694 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:28.436681 7145 raft_consensus.cc:3037] T 72e090a6c18e4a13bbb343eb40a07694 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:28.437625 7145 raft_consensus.cc:2454] T 72e090a6c18e4a13bbb343eb40a07694 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:28.437625 7275 raft_consensus.cc:2454] T 72e090a6c18e4a13bbb343eb40a07694 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:28.437950 6817 leader_election.cc:302] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:28.438174 8397 raft_consensus.cc:2781] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:28.438410 8397 raft_consensus.cc:686] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 01:59:28.438586 8397 consensus_queue.cc:237] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:28.440564 6740 catalog_manager.cc:5484] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:28.607143 8079 raft_consensus.cc:482] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:28.607278 8079 raft_consensus.cc:504] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:28.607697 8079 leader_election.cc:288] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:28.608143 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4600418bf64c4a0d962831c6924c46e1" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:28.608150 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4600418bf64c4a0d962831c6924c46e1" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:28.608285 7143 raft_consensus.cc:2454] T 4600418bf64c4a0d962831c6924c46e1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:28.608279 6881 raft_consensus.cc:2454] T 4600418bf64c4a0d962831c6924c46e1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:28.608582 7212 leader_election.cc:302] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:28.608732 8079 raft_consensus.cc:2781] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:28.608786 8079 raft_consensus.cc:482] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:28.608826 8079 raft_consensus.cc:3037] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:28.609807 8079 raft_consensus.cc:504] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:28.610175 8079 leader_election.cc:288] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Requested vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:28.610618 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4600418bf64c4a0d962831c6924c46e1" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:28.610730 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4600418bf64c4a0d962831c6924c46e1" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:28.610777 6881 raft_consensus.cc:3037] T 4600418bf64c4a0d962831c6924c46e1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:28.610864 7143 raft_consensus.cc:3037] T 4600418bf64c4a0d962831c6924c46e1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:28.611582 6881 raft_consensus.cc:2454] T 4600418bf64c4a0d962831c6924c46e1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:28.611639 7143 raft_consensus.cc:2454] T 4600418bf64c4a0d962831c6924c46e1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:28.611888 7212 leader_election.cc:302] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:28.612107 8079 raft_consensus.cc:2781] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:28.612193 8079 raft_consensus.cc:686] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 01:59:28.612341 8079 consensus_queue.cc:237] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:28.614756 6740 catalog_manager.cc:5484] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
W20240503 01:59:28.759887 8527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:28.760190 8527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:28.763523 8527 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240503 01:59:28.819218 8364 consensus_queue.cc:1035] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:28.820420 8397 consensus_queue.cc:1035] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:28.826498 8524 consensus_queue.cc:1035] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:28.841256 8522 consensus_queue.cc:1035] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:28.881904 6881 raft_consensus.cc:1261] T 4600418bf64c4a0d962831c6924c46e1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:28.881915 7143 raft_consensus.cc:1261] T 4600418bf64c4a0d962831c6924c46e1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:28.882333 8079 consensus_queue.cc:1035] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:28.882674 8228 consensus_queue.cc:1035] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 148.565 ms
time per row: 0.0742824 ms
I20240503 01:59:28.971679 6738 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:28.979583 6993 tablet_service.cc:1467] Processing CreateTablet for tablet 49b580d648014ce899fee8b3b329c47d (DEFAULT_TABLE table=rebalance_test_table_10 [id=f8bdc24361c34c13ac0d377e11dc39ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:28.979856 6994 tablet_service.cc:1467] Processing CreateTablet for tablet ccbd932e20b44dc5aceb86c86d7fda6b (DEFAULT_TABLE table=rebalance_test_table_10 [id=f8bdc24361c34c13ac0d377e11dc39ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:28.979950 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49b580d648014ce899fee8b3b329c47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:28.980082 6994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccbd932e20b44dc5aceb86c86d7fda6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:28.980643 7123 tablet_service.cc:1467] Processing CreateTablet for tablet ccbd932e20b44dc5aceb86c86d7fda6b (DEFAULT_TABLE table=rebalance_test_table_10 [id=f8bdc24361c34c13ac0d377e11dc39ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:28.980652 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 49b580d648014ce899fee8b3b329c47d (DEFAULT_TABLE table=rebalance_test_table_10 [id=f8bdc24361c34c13ac0d377e11dc39ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:28.980902 7123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccbd932e20b44dc5aceb86c86d7fda6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:28.980913 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49b580d648014ce899fee8b3b329c47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:28.982658 6863 tablet_service.cc:1467] Processing CreateTablet for tablet 684d38c6115a4d22855bb90b0b784a1f (DEFAULT_TABLE table=rebalance_test_table_10 [id=f8bdc24361c34c13ac0d377e11dc39ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:28.982932 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 684d38c6115a4d22855bb90b0b784a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:28.983227 8577 tablet_bootstrap.cc:492] T ccbd932e20b44dc5aceb86c86d7fda6b P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:28.984041 8579 tablet_bootstrap.cc:492] T 49b580d648014ce899fee8b3b329c47d P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:28.984953 8578 tablet_bootstrap.cc:492] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:28.984992 8579 tablet_bootstrap.cc:654] T 49b580d648014ce899fee8b3b329c47d P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:28.984917 7256 tablet_service.cc:1467] Processing CreateTablet for tablet 49b580d648014ce899fee8b3b329c47d (DEFAULT_TABLE table=rebalance_test_table_10 [id=f8bdc24361c34c13ac0d377e11dc39ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:28.985211 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49b580d648014ce899fee8b3b329c47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:28.985965 8578 tablet_bootstrap.cc:654] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:28.986121 6993 tablet_service.cc:1467] Processing CreateTablet for tablet 684d38c6115a4d22855bb90b0b784a1f (DEFAULT_TABLE table=rebalance_test_table_10 [id=f8bdc24361c34c13ac0d377e11dc39ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:28.986372 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 684d38c6115a4d22855bb90b0b784a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:28.987099 8578 tablet_bootstrap.cc:492] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:28.987187 8578 ts_tablet_manager.cc:1367] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:28.987613 8582 tablet_bootstrap.cc:492] T 49b580d648014ce899fee8b3b329c47d P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:28.987684 8578 raft_consensus.cc:348] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:28.987793 8578 raft_consensus.cc:374] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:28.987826 8578 raft_consensus.cc:729] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:28.987790 7254 tablet_service.cc:1467] Processing CreateTablet for tablet ccbd932e20b44dc5aceb86c86d7fda6b (DEFAULT_TABLE table=rebalance_test_table_10 [id=f8bdc24361c34c13ac0d377e11dc39ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:28.987965 8578 consensus_queue.cc:260] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:28.988021 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccbd932e20b44dc5aceb86c86d7fda6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:28.988245 8578 ts_tablet_manager.cc:1398] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:28.988900 8582 tablet_bootstrap.cc:654] T 49b580d648014ce899fee8b3b329c47d P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:28.989012 8579 tablet_bootstrap.cc:492] T 49b580d648014ce899fee8b3b329c47d P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:28.989099 8579 ts_tablet_manager.cc:1367] T 49b580d648014ce899fee8b3b329c47d P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:28.989923 8579 raft_consensus.cc:348] T 49b580d648014ce899fee8b3b329c47d P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:28.990034 8579 raft_consensus.cc:374] T 49b580d648014ce899fee8b3b329c47d P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:28.990072 8579 raft_consensus.cc:729] T 49b580d648014ce899fee8b3b329c47d P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:28.990211 7255 tablet_service.cc:1467] Processing CreateTablet for tablet 684d38c6115a4d22855bb90b0b784a1f (DEFAULT_TABLE table=rebalance_test_table_10 [id=f8bdc24361c34c13ac0d377e11dc39ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:28.990211 8579 consensus_queue.cc:260] T 49b580d648014ce899fee8b3b329c47d P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:28.990463 7255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 684d38c6115a4d22855bb90b0b784a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:28.990584 8579 ts_tablet_manager.cc:1398] T 49b580d648014ce899fee8b3b329c47d P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:28.990734 8579 tablet_bootstrap.cc:492] T 684d38c6115a4d22855bb90b0b784a1f P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:28.991523 8579 tablet_bootstrap.cc:654] T 684d38c6115a4d22855bb90b0b784a1f P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:28.992431 8579 tablet_bootstrap.cc:492] T 684d38c6115a4d22855bb90b0b784a1f P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:28.992518 8579 ts_tablet_manager.cc:1367] T 684d38c6115a4d22855bb90b0b784a1f P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:28.992977 8579 raft_consensus.cc:348] T 684d38c6115a4d22855bb90b0b784a1f P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:28.993083 8579 raft_consensus.cc:374] T 684d38c6115a4d22855bb90b0b784a1f P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:28.993111 8579 raft_consensus.cc:729] T 684d38c6115a4d22855bb90b0b784a1f P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:28.993253 8579 consensus_queue.cc:260] T 684d38c6115a4d22855bb90b0b784a1f P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:28.993542 8579 ts_tablet_manager.cc:1398] T 684d38c6115a4d22855bb90b0b784a1f P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:28.984246 8577 tablet_bootstrap.cc:654] T ccbd932e20b44dc5aceb86c86d7fda6b P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:28.995528 8577 tablet_bootstrap.cc:492] T ccbd932e20b44dc5aceb86c86d7fda6b P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:28.995637 8577 ts_tablet_manager.cc:1367] T ccbd932e20b44dc5aceb86c86d7fda6b P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 01:59:28.996037 8577 raft_consensus.cc:348] T ccbd932e20b44dc5aceb86c86d7fda6b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:28.996145 8577 raft_consensus.cc:374] T ccbd932e20b44dc5aceb86c86d7fda6b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:28.996186 8577 raft_consensus.cc:729] T ccbd932e20b44dc5aceb86c86d7fda6b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:28.996344 8577 consensus_queue.cc:260] T ccbd932e20b44dc5aceb86c86d7fda6b P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:28.996650 8577 ts_tablet_manager.cc:1398] T ccbd932e20b44dc5aceb86c86d7fda6b P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:28.996855 8577 tablet_bootstrap.cc:492] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:28.997877 8577 tablet_bootstrap.cc:654] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:28.999887 8582 tablet_bootstrap.cc:492] T 49b580d648014ce899fee8b3b329c47d P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:29.000011 8582 ts_tablet_manager.cc:1367] T 49b580d648014ce899fee8b3b329c47d P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 01:59:29.000031 8577 tablet_bootstrap.cc:492] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:29.000128 8577 ts_tablet_manager.cc:1367] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:29.000726 8577 raft_consensus.cc:348] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:29.000828 8577 raft_consensus.cc:374] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.000560 8582 raft_consensus.cc:348] T 49b580d648014ce899fee8b3b329c47d P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:29.000850 8577 raft_consensus.cc:729] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.000873 8582 raft_consensus.cc:374] T 49b580d648014ce899fee8b3b329c47d P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.000927 8582 raft_consensus.cc:729] T 49b580d648014ce899fee8b3b329c47d P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.000990 8577 consensus_queue.cc:260] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:29.001088 8582 consensus_queue.cc:260] T 49b580d648014ce899fee8b3b329c47d P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:29.001297 8577 ts_tablet_manager.cc:1398] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:29.001350 8582 ts_tablet_manager.cc:1398] T 49b580d648014ce899fee8b3b329c47d P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:29.001472 8577 tablet_bootstrap.cc:492] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:29.001540 8582 tablet_bootstrap.cc:492] T ccbd932e20b44dc5aceb86c86d7fda6b P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:29.002455 8582 tablet_bootstrap.cc:654] T ccbd932e20b44dc5aceb86c86d7fda6b P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:29.002647 8577 tablet_bootstrap.cc:654] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:29.004287 8582 tablet_bootstrap.cc:492] T ccbd932e20b44dc5aceb86c86d7fda6b P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:29.004377 8582 ts_tablet_manager.cc:1367] T ccbd932e20b44dc5aceb86c86d7fda6b P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:29.004848 8582 raft_consensus.cc:348] T ccbd932e20b44dc5aceb86c86d7fda6b P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:29.004967 8582 raft_consensus.cc:374] T ccbd932e20b44dc5aceb86c86d7fda6b P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.004966 8577 tablet_bootstrap.cc:492] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:29.005000 8582 raft_consensus.cc:729] T ccbd932e20b44dc5aceb86c86d7fda6b P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.005052 8577 ts_tablet_manager.cc:1367] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 01:59:29.005132 8582 consensus_queue.cc:260] T ccbd932e20b44dc5aceb86c86d7fda6b P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:29.005467 8582 ts_tablet_manager.cc:1398] T ccbd932e20b44dc5aceb86c86d7fda6b P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:29.005506 8577 raft_consensus.cc:348] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:29.005622 8577 raft_consensus.cc:374] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.005652 8577 raft_consensus.cc:729] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.005791 8577 consensus_queue.cc:260] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:29.005903 8582 tablet_bootstrap.cc:492] T 684d38c6115a4d22855bb90b0b784a1f P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:29.006072 8577 ts_tablet_manager.cc:1398] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:29.006887 8582 tablet_bootstrap.cc:654] T 684d38c6115a4d22855bb90b0b784a1f P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:29.007866 8582 tablet_bootstrap.cc:492] T 684d38c6115a4d22855bb90b0b784a1f P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:29.007961 8582 ts_tablet_manager.cc:1367] T 684d38c6115a4d22855bb90b0b784a1f P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:29.008355 8582 raft_consensus.cc:348] T 684d38c6115a4d22855bb90b0b784a1f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:29.008451 8582 raft_consensus.cc:374] T 684d38c6115a4d22855bb90b0b784a1f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.008472 8582 raft_consensus.cc:729] T 684d38c6115a4d22855bb90b0b784a1f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.008572 8582 consensus_queue.cc:260] T 684d38c6115a4d22855bb90b0b784a1f P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:29.008838 8582 ts_tablet_manager.cc:1398] T 684d38c6115a4d22855bb90b0b784a1f P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:29.056959 8112 raft_consensus.cc:482] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:29.057121 8112 raft_consensus.cc:504] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:29.057608 8112 leader_election.cc:288] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:29.058040 7276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccbd932e20b44dc5aceb86c86d7fda6b" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:29.058099 7013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccbd932e20b44dc5aceb86c86d7fda6b" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:29.058178 7276 raft_consensus.cc:2454] T ccbd932e20b44dc5aceb86c86d7fda6b P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:29.058255 7013 raft_consensus.cc:2454] T ccbd932e20b44dc5aceb86c86d7fda6b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:29.058610 7079 leader_election.cc:302] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:29.058776 8112 raft_consensus.cc:2781] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:29.058851 8112 raft_consensus.cc:482] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:29.058888 8112 raft_consensus.cc:3037] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.059962 8112 raft_consensus.cc:504] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:29.060308 8112 leader_election.cc:288] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:29.060703 7276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccbd932e20b44dc5aceb86c86d7fda6b" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:29.060714 7013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccbd932e20b44dc5aceb86c86d7fda6b" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:29.060832 7276 raft_consensus.cc:3037] T ccbd932e20b44dc5aceb86c86d7fda6b P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.060848 7013 raft_consensus.cc:3037] T ccbd932e20b44dc5aceb86c86d7fda6b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.061684 7276 raft_consensus.cc:2454] T ccbd932e20b44dc5aceb86c86d7fda6b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:29.061684 7013 raft_consensus.cc:2454] T ccbd932e20b44dc5aceb86c86d7fda6b P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:29.062011 7078 leader_election.cc:302] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:29.062206 8112 raft_consensus.cc:2781] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:29.062419 8112 raft_consensus.cc:686] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Becoming Leader. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Running, Role: LEADER
I20240503 01:59:29.062611 8112 consensus_queue.cc:237] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:29.064803 6741 catalog_manager.cc:5484] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:29.081604 8364 raft_consensus.cc:482] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:29.081758 8364 raft_consensus.cc:504] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:29.082191 8364 leader_election.cc:288] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:29.082717 7276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b580d648014ce899fee8b3b329c47d" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:29.082716 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b580d648014ce899fee8b3b329c47d" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:29.082895 6881 raft_consensus.cc:2454] T 49b580d648014ce899fee8b3b329c47d P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:29.082890 7276 raft_consensus.cc:2454] T 49b580d648014ce899fee8b3b329c47d P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:29.083217 6950 leader_election.cc:302] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:29.083498 8364 raft_consensus.cc:2781] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:29.083559 8364 raft_consensus.cc:482] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:29.083602 8364 raft_consensus.cc:3037] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.084746 8364 raft_consensus.cc:504] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:29.085208 8364 leader_election.cc:288] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:29.085458 7276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b580d648014ce899fee8b3b329c47d" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:29.085580 7276 raft_consensus.cc:3037] T 49b580d648014ce899fee8b3b329c47d P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.085597 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49b580d648014ce899fee8b3b329c47d" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:29.085711 6881 raft_consensus.cc:3037] T 49b580d648014ce899fee8b3b329c47d P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.086477 6881 raft_consensus.cc:2454] T 49b580d648014ce899fee8b3b329c47d P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:29.086582 7276 raft_consensus.cc:2454] T 49b580d648014ce899fee8b3b329c47d P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:29.086808 6950 leader_election.cc:302] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:29.086968 8364 raft_consensus.cc:2781] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:29.087054 8364 raft_consensus.cc:686] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:29.087201 8364 consensus_queue.cc:237] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:29.089354 6741 catalog_manager.cc:5484] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:29.255224 8522 raft_consensus.cc:482] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:29.255367 8522 raft_consensus.cc:504] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:29.255816 8522 leader_election.cc:288] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:29.256325 7276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "684d38c6115a4d22855bb90b0b784a1f" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:29.256337 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "684d38c6115a4d22855bb90b0b784a1f" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:29.256474 6881 raft_consensus.cc:2454] T 684d38c6115a4d22855bb90b0b784a1f P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:29.256477 7276 raft_consensus.cc:2454] T 684d38c6115a4d22855bb90b0b784a1f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:29.256811 6948 leader_election.cc:302] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:29.256984 8522 raft_consensus.cc:2781] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:29.257038 8522 raft_consensus.cc:482] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:29.257064 8522 raft_consensus.cc:3037] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.257939 8522 raft_consensus.cc:504] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:29.258296 8522 leader_election.cc:288] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:29.258735 7276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "684d38c6115a4d22855bb90b0b784a1f" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:29.258738 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "684d38c6115a4d22855bb90b0b784a1f" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:29.258848 7276 raft_consensus.cc:3037] T 684d38c6115a4d22855bb90b0b784a1f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.258858 6881 raft_consensus.cc:3037] T 684d38c6115a4d22855bb90b0b784a1f P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.259614 7276 raft_consensus.cc:2454] T 684d38c6115a4d22855bb90b0b784a1f P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:29.259625 6881 raft_consensus.cc:2454] T 684d38c6115a4d22855bb90b0b784a1f P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:29.259938 6948 leader_election.cc:302] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:29.260339 8522 raft_consensus.cc:2781] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:29.260522 8522 raft_consensus.cc:686] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:29.260733 8522 consensus_queue.cc:237] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:29.263316 6741 catalog_manager.cc:5484] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
W20240503 01:59:29.401598 8594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:29.401885 8594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:29.405632 8594 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240503 01:59:29.464411 8112 consensus_queue.cc:1035] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:29.471333 8590 consensus_queue.cc:1035] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:29.480996 8522 consensus_queue.cc:1035] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:29.484660 8364 consensus_queue.cc:1035] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:29.521346 6882 raft_consensus.cc:1261] T 684d38c6115a4d22855bb90b0b784a1f P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:29.521795 8522 consensus_queue.cc:1035] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:29.523265 7275 raft_consensus.cc:1261] T 684d38c6115a4d22855bb90b0b784a1f P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:29.524627 8364 consensus_queue.cc:1035] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 132.816 ms
time per row: 0.0664079 ms
I20240503 01:59:29.591866 6741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:29.598135 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 8295ea038a88485095544cc608b000c2 (DEFAULT_TABLE table=rebalance_test_table_11 [id=dce2b86d058140cfb0a0d5088d32ec71]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:29.598284 7123 tablet_service.cc:1467] Processing CreateTablet for tablet 8295ea038a88485095544cc608b000c2 (DEFAULT_TABLE table=rebalance_test_table_11 [id=dce2b86d058140cfb0a0d5088d32ec71]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:29.598486 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8295ea038a88485095544cc608b000c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:29.598559 7123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8295ea038a88485095544cc608b000c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:29.599092 7255 tablet_service.cc:1467] Processing CreateTablet for tablet 96dd9b1f85cf471495dff16d5fa9148a (DEFAULT_TABLE table=rebalance_test_table_11 [id=dce2b86d058140cfb0a0d5088d32ec71]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:29.599395 7255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96dd9b1f85cf471495dff16d5fa9148a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:29.599092 7254 tablet_service.cc:1467] Processing CreateTablet for tablet dcc773c6d5794e26a0003c7d4acc5d21 (DEFAULT_TABLE table=rebalance_test_table_11 [id=dce2b86d058140cfb0a0d5088d32ec71]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:29.599797 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc773c6d5794e26a0003c7d4acc5d21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:29.601044 6993 tablet_service.cc:1467] Processing CreateTablet for tablet 8295ea038a88485095544cc608b000c2 (DEFAULT_TABLE table=rebalance_test_table_11 [id=dce2b86d058140cfb0a0d5088d32ec71]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:29.601312 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8295ea038a88485095544cc608b000c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:29.601269 6863 tablet_service.cc:1467] Processing CreateTablet for tablet dcc773c6d5794e26a0003c7d4acc5d21 (DEFAULT_TABLE table=rebalance_test_table_11 [id=dce2b86d058140cfb0a0d5088d32ec71]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:29.601506 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc773c6d5794e26a0003c7d4acc5d21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:29.601905 8656 tablet_bootstrap.cc:492] T dcc773c6d5794e26a0003c7d4acc5d21 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:29.601044 6994 tablet_service.cc:1467] Processing CreateTablet for tablet 96dd9b1f85cf471495dff16d5fa9148a (DEFAULT_TABLE table=rebalance_test_table_11 [id=dce2b86d058140cfb0a0d5088d32ec71]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:29.602797 6994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96dd9b1f85cf471495dff16d5fa9148a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:29.603395 8656 tablet_bootstrap.cc:654] T dcc773c6d5794e26a0003c7d4acc5d21 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:29.604298 8654 tablet_bootstrap.cc:492] T 8295ea038a88485095544cc608b000c2 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:29.604338 7124 tablet_service.cc:1467] Processing CreateTablet for tablet dcc773c6d5794e26a0003c7d4acc5d21 (DEFAULT_TABLE table=rebalance_test_table_11 [id=dce2b86d058140cfb0a0d5088d32ec71]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:29.604620 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc773c6d5794e26a0003c7d4acc5d21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:29.605737 8654 tablet_bootstrap.cc:654] T 8295ea038a88485095544cc608b000c2 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:29.606340 8657 tablet_bootstrap.cc:492] T 8295ea038a88485095544cc608b000c2 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:29.606473 8655 tablet_bootstrap.cc:492] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:29.604338 7123 tablet_service.cc:1467] Processing CreateTablet for tablet 96dd9b1f85cf471495dff16d5fa9148a (DEFAULT_TABLE table=rebalance_test_table_11 [id=dce2b86d058140cfb0a0d5088d32ec71]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:29.607287 8657 tablet_bootstrap.cc:654] T 8295ea038a88485095544cc608b000c2 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:29.607429 7123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96dd9b1f85cf471495dff16d5fa9148a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:29.607456 8655 tablet_bootstrap.cc:654] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:29.608326 8654 tablet_bootstrap.cc:492] T 8295ea038a88485095544cc608b000c2 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:29.608393 8654 ts_tablet_manager.cc:1367] T 8295ea038a88485095544cc608b000c2 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:29.608752 8654 raft_consensus.cc:348] T 8295ea038a88485095544cc608b000c2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:29.608846 8654 raft_consensus.cc:374] T 8295ea038a88485095544cc608b000c2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.608877 8654 raft_consensus.cc:729] T 8295ea038a88485095544cc608b000c2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.609004 8654 consensus_queue.cc:260] T 8295ea038a88485095544cc608b000c2 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:29.609272 8654 ts_tablet_manager.cc:1398] T 8295ea038a88485095544cc608b000c2 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:29.610473 8654 tablet_bootstrap.cc:492] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:29.610932 8657 tablet_bootstrap.cc:492] T 8295ea038a88485095544cc608b000c2 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:29.611061 8657 ts_tablet_manager.cc:1367] T 8295ea038a88485095544cc608b000c2 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:29.611514 8654 tablet_bootstrap.cc:654] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:29.611732 8655 tablet_bootstrap.cc:492] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:29.611855 8655 ts_tablet_manager.cc:1367] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 01:59:29.611898 8657 raft_consensus.cc:348] T 8295ea038a88485095544cc608b000c2 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:29.612037 8657 raft_consensus.cc:374] T 8295ea038a88485095544cc608b000c2 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.612087 8657 raft_consensus.cc:729] T 8295ea038a88485095544cc608b000c2 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.612756 8656 tablet_bootstrap.cc:492] T dcc773c6d5794e26a0003c7d4acc5d21 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:29.612869 8656 ts_tablet_manager.cc:1367] T dcc773c6d5794e26a0003c7d4acc5d21 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 01:59:29.612771 8655 raft_consensus.cc:348] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:29.612977 8655 raft_consensus.cc:374] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.613032 8655 raft_consensus.cc:729] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.613193 8655 consensus_queue.cc:260] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:29.613376 8656 raft_consensus.cc:348] T dcc773c6d5794e26a0003c7d4acc5d21 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:29.613494 8656 raft_consensus.cc:374] T dcc773c6d5794e26a0003c7d4acc5d21 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.613528 8656 raft_consensus.cc:729] T dcc773c6d5794e26a0003c7d4acc5d21 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.613641 8656 consensus_queue.cc:260] T dcc773c6d5794e26a0003c7d4acc5d21 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:29.613674 8655 ts_tablet_manager.cc:1398] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:29.613663 8657 consensus_queue.cc:260] T 8295ea038a88485095544cc608b000c2 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:29.613835 8655 tablet_bootstrap.cc:492] T dcc773c6d5794e26a0003c7d4acc5d21 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:29.613893 8656 ts_tablet_manager.cc:1398] T dcc773c6d5794e26a0003c7d4acc5d21 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:29.614069 8656 tablet_bootstrap.cc:492] T 96dd9b1f85cf471495dff16d5fa9148a P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:29.614084 8657 ts_tablet_manager.cc:1398] T 8295ea038a88485095544cc608b000c2 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:29.614243 8657 tablet_bootstrap.cc:492] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:29.614944 8656 tablet_bootstrap.cc:654] T 96dd9b1f85cf471495dff16d5fa9148a P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:29.615370 8657 tablet_bootstrap.cc:654] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:29.615494 8655 tablet_bootstrap.cc:654] T dcc773c6d5794e26a0003c7d4acc5d21 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:29.616709 8657 tablet_bootstrap.cc:492] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:29.616804 8657 ts_tablet_manager.cc:1367] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:29.617405 8657 raft_consensus.cc:348] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:29.617542 8657 raft_consensus.cc:374] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.617586 8657 raft_consensus.cc:729] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.617731 8657 consensus_queue.cc:260] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:29.618119 8657 ts_tablet_manager.cc:1398] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:29.618299 8655 tablet_bootstrap.cc:492] T dcc773c6d5794e26a0003c7d4acc5d21 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:29.618435 8655 ts_tablet_manager.cc:1367] T dcc773c6d5794e26a0003c7d4acc5d21 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 01:59:29.619223 8656 tablet_bootstrap.cc:492] T 96dd9b1f85cf471495dff16d5fa9148a P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:29.619315 8656 ts_tablet_manager.cc:1367] T 96dd9b1f85cf471495dff16d5fa9148a P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 01:59:29.619782 8656 raft_consensus.cc:348] T 96dd9b1f85cf471495dff16d5fa9148a P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:29.619907 8656 raft_consensus.cc:374] T 96dd9b1f85cf471495dff16d5fa9148a P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.619948 8656 raft_consensus.cc:729] T 96dd9b1f85cf471495dff16d5fa9148a P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.619782 8655 raft_consensus.cc:348] T dcc773c6d5794e26a0003c7d4acc5d21 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:29.620095 8655 raft_consensus.cc:374] T dcc773c6d5794e26a0003c7d4acc5d21 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.620118 8656 consensus_queue.cc:260] T 96dd9b1f85cf471495dff16d5fa9148a P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:29.620174 8655 raft_consensus.cc:729] T dcc773c6d5794e26a0003c7d4acc5d21 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.620398 8655 consensus_queue.cc:260] T dcc773c6d5794e26a0003c7d4acc5d21 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:29.620558 8656 ts_tablet_manager.cc:1398] T 96dd9b1f85cf471495dff16d5fa9148a P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:29.620880 8655 ts_tablet_manager.cc:1398] T dcc773c6d5794e26a0003c7d4acc5d21 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:29.621071 8655 tablet_bootstrap.cc:492] T 96dd9b1f85cf471495dff16d5fa9148a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:29.621160 8654 tablet_bootstrap.cc:492] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:29.621258 8654 ts_tablet_manager.cc:1367] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 01:59:29.621752 8654 raft_consensus.cc:348] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:29.621877 8654 raft_consensus.cc:374] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.621917 8654 raft_consensus.cc:729] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.622047 8654 consensus_queue.cc:260] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:29.622749 8655 tablet_bootstrap.cc:654] T 96dd9b1f85cf471495dff16d5fa9148a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:29.623281 8654 ts_tablet_manager.cc:1398] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:29.624496 8655 tablet_bootstrap.cc:492] T 96dd9b1f85cf471495dff16d5fa9148a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:29.624588 8655 ts_tablet_manager.cc:1367] T 96dd9b1f85cf471495dff16d5fa9148a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:29.625049 8655 raft_consensus.cc:348] T 96dd9b1f85cf471495dff16d5fa9148a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:29.625177 8655 raft_consensus.cc:374] T 96dd9b1f85cf471495dff16d5fa9148a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:29.625224 8655 raft_consensus.cc:729] T 96dd9b1f85cf471495dff16d5fa9148a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:29.625357 8655 consensus_queue.cc:260] T 96dd9b1f85cf471495dff16d5fa9148a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:29.625665 8655 ts_tablet_manager.cc:1398] T 96dd9b1f85cf471495dff16d5fa9148a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:29.626029 8642 raft_consensus.cc:482] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:29.626133 8642 raft_consensus.cc:504] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:29.626614 8642 leader_election.cc:288] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:29.626957 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dd9b1f85cf471495dff16d5fa9148a" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:29.627060 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dd9b1f85cf471495dff16d5fa9148a" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:29.627113 7275 raft_consensus.cc:2454] T 96dd9b1f85cf471495dff16d5fa9148a P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:29.627177 7145 raft_consensus.cc:2454] T 96dd9b1f85cf471495dff16d5fa9148a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:29.627419 6948 leader_election.cc:302] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:29.627952 8642 raft_consensus.cc:2781] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:29.628015 8642 raft_consensus.cc:482] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:29.628042 8642 raft_consensus.cc:3037] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.628932 8642 raft_consensus.cc:504] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:29.629266 8642 leader_election.cc:288] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:29.629570 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dd9b1f85cf471495dff16d5fa9148a" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:29.629572 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96dd9b1f85cf471495dff16d5fa9148a" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:29.629683 7275 raft_consensus.cc:3037] T 96dd9b1f85cf471495dff16d5fa9148a P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.629698 7145 raft_consensus.cc:3037] T 96dd9b1f85cf471495dff16d5fa9148a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.630712 7145 raft_consensus.cc:2454] T 96dd9b1f85cf471495dff16d5fa9148a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:29.630712 7275 raft_consensus.cc:2454] T 96dd9b1f85cf471495dff16d5fa9148a P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:29.631040 6949 leader_election.cc:302] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:29.631275 8642 raft_consensus.cc:2781] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:29.631347 8642 raft_consensus.cc:686] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:29.631510 8642 consensus_queue.cc:237] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:29.632190 8647 raft_consensus.cc:482] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:29.632295 8647 raft_consensus.cc:504] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:29.632704 8647 leader_election.cc:288] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:29.633080 7013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8295ea038a88485095544cc608b000c2" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:29.633203 7013 raft_consensus.cc:2454] T 8295ea038a88485095544cc608b000c2 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:29.633195 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8295ea038a88485095544cc608b000c2" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:29.633332 6881 raft_consensus.cc:2454] T 8295ea038a88485095544cc608b000c2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:29.633633 7081 leader_election.cc:302] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:29.633812 8647 raft_consensus.cc:2781] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:29.633878 8647 raft_consensus.cc:482] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:29.633935 8647 raft_consensus.cc:3037] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.634343 6738 catalog_manager.cc:5484] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:29.634802 8647 raft_consensus.cc:504] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:29.635174 8647 leader_election.cc:288] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Requested vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:29.635526 7013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8295ea038a88485095544cc608b000c2" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:29.635648 7013 raft_consensus.cc:3037] T 8295ea038a88485095544cc608b000c2 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.636631 7013 raft_consensus.cc:2454] T 8295ea038a88485095544cc608b000c2 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:29.636890 7078 leader_election.cc:302] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:29.636930 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8295ea038a88485095544cc608b000c2" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:29.637053 6881 raft_consensus.cc:3037] T 8295ea038a88485095544cc608b000c2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.637063 8647 raft_consensus.cc:2781] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:29.637135 8647 raft_consensus.cc:686] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Becoming Leader. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Running, Role: LEADER
I20240503 01:59:29.637270 8647 consensus_queue.cc:237] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:29.637879 6881 raft_consensus.cc:2454] T 8295ea038a88485095544cc608b000c2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:29.639226 6738 catalog_manager.cc:5484] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:29.861953 8625 raft_consensus.cc:482] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:29.862114 8625 raft_consensus.cc:504] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:29.862551 8625 leader_election.cc:288] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:29.863047 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc773c6d5794e26a0003c7d4acc5d21" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:29.863049 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc773c6d5794e26a0003c7d4acc5d21" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:29.863190 7145 raft_consensus.cc:2454] T dcc773c6d5794e26a0003c7d4acc5d21 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:29.863206 7275 raft_consensus.cc:2454] T dcc773c6d5794e26a0003c7d4acc5d21 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:29.863483 6818 leader_election.cc:302] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:29.863688 8625 raft_consensus.cc:2781] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:29.863761 8625 raft_consensus.cc:482] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:29.863801 8625 raft_consensus.cc:3037] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.864706 8625 raft_consensus.cc:504] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:29.865051 8625 leader_election.cc:288] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Requested vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:29.865392 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc773c6d5794e26a0003c7d4acc5d21" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:29.865510 7145 raft_consensus.cc:3037] T dcc773c6d5794e26a0003c7d4acc5d21 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.865547 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc773c6d5794e26a0003c7d4acc5d21" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:29.865806 7275 raft_consensus.cc:3037] T dcc773c6d5794e26a0003c7d4acc5d21 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:29.866364 7145 raft_consensus.cc:2454] T dcc773c6d5794e26a0003c7d4acc5d21 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:29.866644 7275 raft_consensus.cc:2454] T dcc773c6d5794e26a0003c7d4acc5d21 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:29.866719 6818 leader_election.cc:302] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:29.866948 8625 raft_consensus.cc:2781] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:29.867089 8625 raft_consensus.cc:686] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 01:59:29.867255 8625 consensus_queue.cc:237] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:29.869205 6738 catalog_manager.cc:5484] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
W20240503 01:59:30.021616 8670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:30.021891 8670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:30.025344 8670 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240503 01:59:30.121078 7275 raft_consensus.cc:1261] T 96dd9b1f85cf471495dff16d5fa9148a P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:30.121112 7145 raft_consensus.cc:1261] T 96dd9b1f85cf471495dff16d5fa9148a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:30.121505 8642 consensus_queue.cc:1035] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:30.121826 8641 consensus_queue.cc:1035] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:30.127147 7013 raft_consensus.cc:1261] T 8295ea038a88485095544cc608b000c2 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:30.127811 6881 raft_consensus.cc:1261] T 8295ea038a88485095544cc608b000c2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:30.128207 8647 consensus_queue.cc:1035] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:30.128489 8648 consensus_queue.cc:1035] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:30.161206 7275 raft_consensus.cc:1261] T dcc773c6d5794e26a0003c7d4acc5d21 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:30.161286 7145 raft_consensus.cc:1261] T dcc773c6d5794e26a0003c7d4acc5d21 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:30.161723 8625 consensus_queue.cc:1035] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:30.162063 8397 consensus_queue.cc:1035] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 134.913 ms
time per row: 0.0674565 ms
I20240503 01:59:30.209201 6741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:30.215541 6863 tablet_service.cc:1467] Processing CreateTablet for tablet 1eca51b6a18e4718b43f318ed3147338 (DEFAULT_TABLE table=rebalance_test_table_12 [id=9e4998e19d924c9a9fd0a10e74520d26]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.215673 7123 tablet_service.cc:1467] Processing CreateTablet for tablet 1eca51b6a18e4718b43f318ed3147338 (DEFAULT_TABLE table=rebalance_test_table_12 [id=9e4998e19d924c9a9fd0a10e74520d26]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.215930 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eca51b6a18e4718b43f318ed3147338. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.215977 7123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eca51b6a18e4718b43f318ed3147338. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.216073 7124 tablet_service.cc:1467] Processing CreateTablet for tablet 789162c6428c49fb9c0edd30960d13c1 (DEFAULT_TABLE table=rebalance_test_table_12 [id=9e4998e19d924c9a9fd0a10e74520d26]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.216325 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 789162c6428c49fb9c0edd30960d13c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.216357 7254 tablet_service.cc:1467] Processing CreateTablet for tablet f827ec6764cc4dccba77493b3d1d8ec7 (DEFAULT_TABLE table=rebalance_test_table_12 [id=9e4998e19d924c9a9fd0a10e74520d26]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.216621 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f827ec6764cc4dccba77493b3d1d8ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.217857 6994 tablet_service.cc:1467] Processing CreateTablet for tablet 1eca51b6a18e4718b43f318ed3147338 (DEFAULT_TABLE table=rebalance_test_table_12 [id=9e4998e19d924c9a9fd0a10e74520d26]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.218149 6994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eca51b6a18e4718b43f318ed3147338. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.218978 7255 tablet_service.cc:1467] Processing CreateTablet for tablet 789162c6428c49fb9c0edd30960d13c1 (DEFAULT_TABLE table=rebalance_test_table_12 [id=9e4998e19d924c9a9fd0a10e74520d26]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.219182 8730 tablet_bootstrap.cc:492] T f827ec6764cc4dccba77493b3d1d8ec7 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:30.219250 7255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 789162c6428c49fb9c0edd30960d13c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.220630 6861 tablet_service.cc:1467] Processing CreateTablet for tablet f827ec6764cc4dccba77493b3d1d8ec7 (DEFAULT_TABLE table=rebalance_test_table_12 [id=9e4998e19d924c9a9fd0a10e74520d26]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.220901 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f827ec6764cc4dccba77493b3d1d8ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.221154 8730 tablet_bootstrap.cc:654] T f827ec6764cc4dccba77493b3d1d8ec7 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.221385 8729 tablet_bootstrap.cc:492] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:30.222213 8729 tablet_bootstrap.cc:654] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.223177 6993 tablet_service.cc:1467] Processing CreateTablet for tablet 789162c6428c49fb9c0edd30960d13c1 (DEFAULT_TABLE table=rebalance_test_table_12 [id=9e4998e19d924c9a9fd0a10e74520d26]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.223412 8729 tablet_bootstrap.cc:492] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:30.223459 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 789162c6428c49fb9c0edd30960d13c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.223488 8729 ts_tablet_manager.cc:1367] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:30.223501 8731 tablet_bootstrap.cc:492] T 1eca51b6a18e4718b43f318ed3147338 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:30.223855 8729 raft_consensus.cc:348] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:30.223950 8729 raft_consensus.cc:374] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.223971 8729 raft_consensus.cc:729] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.224090 8729 consensus_queue.cc:260] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:30.224347 8729 ts_tablet_manager.cc:1398] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:30.224356 8731 tablet_bootstrap.cc:654] T 1eca51b6a18e4718b43f318ed3147338 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.224476 8729 tablet_bootstrap.cc:492] T 789162c6428c49fb9c0edd30960d13c1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:30.225267 8729 tablet_bootstrap.cc:654] T 789162c6428c49fb9c0edd30960d13c1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.225553 8731 tablet_bootstrap.cc:492] T 1eca51b6a18e4718b43f318ed3147338 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:30.225637 8731 ts_tablet_manager.cc:1367] T 1eca51b6a18e4718b43f318ed3147338 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:30.226181 8731 raft_consensus.cc:348] T 1eca51b6a18e4718b43f318ed3147338 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:30.226343 8731 raft_consensus.cc:374] T 1eca51b6a18e4718b43f318ed3147338 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.226425 8731 raft_consensus.cc:729] T 1eca51b6a18e4718b43f318ed3147338 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.226670 8731 consensus_queue.cc:260] T 1eca51b6a18e4718b43f318ed3147338 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:30.226933 6994 tablet_service.cc:1467] Processing CreateTablet for tablet f827ec6764cc4dccba77493b3d1d8ec7 (DEFAULT_TABLE table=rebalance_test_table_12 [id=9e4998e19d924c9a9fd0a10e74520d26]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.226989 8732 tablet_bootstrap.cc:492] T 1eca51b6a18e4718b43f318ed3147338 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:30.227172 6994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f827ec6764cc4dccba77493b3d1d8ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.227962 8732 tablet_bootstrap.cc:654] T 1eca51b6a18e4718b43f318ed3147338 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.229269 8731 ts_tablet_manager.cc:1398] T 1eca51b6a18e4718b43f318ed3147338 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 01:59:30.229427 8730 tablet_bootstrap.cc:492] T f827ec6764cc4dccba77493b3d1d8ec7 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:30.229497 8731 tablet_bootstrap.cc:492] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:30.229557 8730 ts_tablet_manager.cc:1367] T f827ec6764cc4dccba77493b3d1d8ec7 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 01:59:30.230078 8730 raft_consensus.cc:348] T f827ec6764cc4dccba77493b3d1d8ec7 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:30.230211 8730 raft_consensus.cc:374] T f827ec6764cc4dccba77493b3d1d8ec7 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.230252 8730 raft_consensus.cc:729] T f827ec6764cc4dccba77493b3d1d8ec7 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.230255 8732 tablet_bootstrap.cc:492] T 1eca51b6a18e4718b43f318ed3147338 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:30.230345 8732 ts_tablet_manager.cc:1367] T 1eca51b6a18e4718b43f318ed3147338 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:30.230367 8730 consensus_queue.cc:260] T f827ec6764cc4dccba77493b3d1d8ec7 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:30.230722 8730 ts_tablet_manager.cc:1398] T f827ec6764cc4dccba77493b3d1d8ec7 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:30.230916 8730 tablet_bootstrap.cc:492] T 789162c6428c49fb9c0edd30960d13c1 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:30.231035 8731 tablet_bootstrap.cc:654] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.231904 8732 raft_consensus.cc:348] T 1eca51b6a18e4718b43f318ed3147338 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:30.232024 8730 tablet_bootstrap.cc:654] T 789162c6428c49fb9c0edd30960d13c1 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.232131 8732 raft_consensus.cc:374] T 1eca51b6a18e4718b43f318ed3147338 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.232224 8732 raft_consensus.cc:729] T 1eca51b6a18e4718b43f318ed3147338 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.232489 8729 tablet_bootstrap.cc:492] T 789162c6428c49fb9c0edd30960d13c1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:30.232594 8729 ts_tablet_manager.cc:1367] T 789162c6428c49fb9c0edd30960d13c1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240503 01:59:30.233062 8729 raft_consensus.cc:348] T 789162c6428c49fb9c0edd30960d13c1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:30.232445 8732 consensus_queue.cc:260] T 1eca51b6a18e4718b43f318ed3147338 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:30.233175 8729 raft_consensus.cc:374] T 789162c6428c49fb9c0edd30960d13c1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.233209 8729 raft_consensus.cc:729] T 789162c6428c49fb9c0edd30960d13c1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.233332 8729 consensus_queue.cc:260] T 789162c6428c49fb9c0edd30960d13c1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:30.233521 8732 ts_tablet_manager.cc:1398] T 1eca51b6a18e4718b43f318ed3147338 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:30.233659 8729 ts_tablet_manager.cc:1398] T 789162c6428c49fb9c0edd30960d13c1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:30.233786 8732 tablet_bootstrap.cc:492] T f827ec6764cc4dccba77493b3d1d8ec7 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:30.234169 8731 tablet_bootstrap.cc:492] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:30.234261 8731 ts_tablet_manager.cc:1367] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 01:59:30.234786 8731 raft_consensus.cc:348] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:30.234933 8731 raft_consensus.cc:374] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.234984 8731 raft_consensus.cc:729] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.235131 8731 consensus_queue.cc:260] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:30.235471 8731 ts_tablet_manager.cc:1398] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:30.235693 8731 tablet_bootstrap.cc:492] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:30.235755 8732 tablet_bootstrap.cc:654] T f827ec6764cc4dccba77493b3d1d8ec7 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.236979 8730 tablet_bootstrap.cc:492] T 789162c6428c49fb9c0edd30960d13c1 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:30.237085 8730 ts_tablet_manager.cc:1367] T 789162c6428c49fb9c0edd30960d13c1 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:30.237417 8731 tablet_bootstrap.cc:654] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.237650 8730 raft_consensus.cc:348] T 789162c6428c49fb9c0edd30960d13c1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:30.237764 8730 raft_consensus.cc:374] T 789162c6428c49fb9c0edd30960d13c1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.237797 8730 raft_consensus.cc:729] T 789162c6428c49fb9c0edd30960d13c1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.237934 8730 consensus_queue.cc:260] T 789162c6428c49fb9c0edd30960d13c1 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:30.238003 8732 tablet_bootstrap.cc:492] T f827ec6764cc4dccba77493b3d1d8ec7 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:30.238083 8732 ts_tablet_manager.cc:1367] T f827ec6764cc4dccba77493b3d1d8ec7 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:30.238248 8730 ts_tablet_manager.cc:1398] T 789162c6428c49fb9c0edd30960d13c1 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:30.238584 8732 raft_consensus.cc:348] T f827ec6764cc4dccba77493b3d1d8ec7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:30.238687 8732 raft_consensus.cc:374] T f827ec6764cc4dccba77493b3d1d8ec7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.238718 8732 raft_consensus.cc:729] T f827ec6764cc4dccba77493b3d1d8ec7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.238924 8732 consensus_queue.cc:260] T f827ec6764cc4dccba77493b3d1d8ec7 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:30.239235 8732 ts_tablet_manager.cc:1398] T f827ec6764cc4dccba77493b3d1d8ec7 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:30.239372 8731 tablet_bootstrap.cc:492] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:30.239477 8731 ts_tablet_manager.cc:1367] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 01:59:30.239930 8731 raft_consensus.cc:348] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:30.240069 8731 raft_consensus.cc:374] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.240119 8731 raft_consensus.cc:729] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.240244 8731 consensus_queue.cc:260] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:30.240556 8731 ts_tablet_manager.cc:1398] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:30.278847 8641 raft_consensus.cc:482] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:30.279044 8641 raft_consensus.cc:504] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:30.279525 8641 leader_election.cc:288] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:30.280063 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f827ec6764cc4dccba77493b3d1d8ec7" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:30.280069 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f827ec6764cc4dccba77493b3d1d8ec7" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:30.280221 7275 raft_consensus.cc:2454] T f827ec6764cc4dccba77493b3d1d8ec7 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:30.280225 6881 raft_consensus.cc:2454] T f827ec6764cc4dccba77493b3d1d8ec7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:30.280558 6948 leader_election.cc:302] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:30.280742 8641 raft_consensus.cc:2781] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:30.280812 8641 raft_consensus.cc:482] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:30.280865 8641 raft_consensus.cc:3037] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.281917 8641 raft_consensus.cc:504] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:30.282289 8641 leader_election.cc:288] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:30.282716 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f827ec6764cc4dccba77493b3d1d8ec7" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:30.282856 6881 raft_consensus.cc:3037] T f827ec6764cc4dccba77493b3d1d8ec7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.282891 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f827ec6764cc4dccba77493b3d1d8ec7" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:30.283109 7275 raft_consensus.cc:3037] T f827ec6764cc4dccba77493b3d1d8ec7 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.283985 6881 raft_consensus.cc:2454] T f827ec6764cc4dccba77493b3d1d8ec7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:30.284421 7275 raft_consensus.cc:2454] T f827ec6764cc4dccba77493b3d1d8ec7 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:30.284407 6950 leader_election.cc:302] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:30.284574 8642 raft_consensus.cc:2781] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:30.284796 8642 raft_consensus.cc:686] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:30.284969 8642 consensus_queue.cc:237] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:30.287240 6738 catalog_manager.cc:5484] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:30.412895 8647 raft_consensus.cc:482] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:30.413077 8647 raft_consensus.cc:504] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:30.413550 8647 leader_election.cc:288] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:30.414098 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eca51b6a18e4718b43f318ed3147338" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:30.414112 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eca51b6a18e4718b43f318ed3147338" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:30.414302 7014 raft_consensus.cc:2454] T 1eca51b6a18e4718b43f318ed3147338 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:30.414319 6881 raft_consensus.cc:2454] T 1eca51b6a18e4718b43f318ed3147338 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:30.414685 7081 leader_election.cc:302] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:30.414908 8647 raft_consensus.cc:2781] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:30.414973 8647 raft_consensus.cc:482] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:30.415009 8647 raft_consensus.cc:3037] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.415915 8647 raft_consensus.cc:504] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:30.416357 8647 leader_election.cc:288] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Requested vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:30.417179 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eca51b6a18e4718b43f318ed3147338" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:30.417250 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eca51b6a18e4718b43f318ed3147338" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:30.417341 7014 raft_consensus.cc:3037] T 1eca51b6a18e4718b43f318ed3147338 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.417362 6881 raft_consensus.cc:3037] T 1eca51b6a18e4718b43f318ed3147338 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.418401 7014 raft_consensus.cc:2454] T 1eca51b6a18e4718b43f318ed3147338 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:30.418401 6881 raft_consensus.cc:2454] T 1eca51b6a18e4718b43f318ed3147338 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:30.418795 7081 leader_election.cc:302] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:30.418996 8647 raft_consensus.cc:2781] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:30.419111 8647 raft_consensus.cc:686] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Becoming Leader. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Running, Role: LEADER
I20240503 01:59:30.419270 8647 consensus_queue.cc:237] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:30.421356 6738 catalog_manager.cc:5484] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:30.560570 8642 raft_consensus.cc:482] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:30.560750 8642 raft_consensus.cc:504] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:30.561224 8642 leader_election.cc:288] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:30.561753 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789162c6428c49fb9c0edd30960d13c1" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:30.561909 7275 raft_consensus.cc:2454] T 789162c6428c49fb9c0edd30960d13c1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:30.561884 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789162c6428c49fb9c0edd30960d13c1" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:30.562238 7145 raft_consensus.cc:2454] T 789162c6428c49fb9c0edd30960d13c1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:30.562446 6948 leader_election.cc:302] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:30.562705 8642 raft_consensus.cc:2781] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:30.562765 8642 raft_consensus.cc:482] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:30.562795 8642 raft_consensus.cc:3037] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.563827 8642 raft_consensus.cc:504] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:30.564230 8642 leader_election.cc:288] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:30.564749 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789162c6428c49fb9c0edd30960d13c1" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:30.564869 7145 raft_consensus.cc:3037] T 789162c6428c49fb9c0edd30960d13c1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.564807 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "789162c6428c49fb9c0edd30960d13c1" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:30.564957 7275 raft_consensus.cc:3037] T 789162c6428c49fb9c0edd30960d13c1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:30.565693 7145 raft_consensus.cc:2454] T 789162c6428c49fb9c0edd30960d13c1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:30.566033 7275 raft_consensus.cc:2454] T 789162c6428c49fb9c0edd30960d13c1 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:30.566381 6948 leader_election.cc:302] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:30.566634 8642 raft_consensus.cc:2781] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:30.566740 8642 raft_consensus.cc:686] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:30.567130 8642 consensus_queue.cc:237] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:30.569655 6738 catalog_manager.cc:5484] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:30.712893 8642 consensus_queue.cc:1035] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:30.716500 8642 consensus_queue.cc:1035] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 01:59:30.723222 8746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:30.723547 8746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:30.727085 8746 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_12'
I20240503 01:59:30.832017 7014 raft_consensus.cc:1261] T 1eca51b6a18e4718b43f318ed3147338 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:30.832485 8648 consensus_queue.cc:1035] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:30.833178 6882 raft_consensus.cc:1261] T 1eca51b6a18e4718b43f318ed3147338 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:30.833830 8648 consensus_queue.cc:1035] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:30.846967 8749 mvcc.cc:205] Tried to move back new op lower bound from 7023417634134446080 to 7023417632438616064. Current Snapshot: MvccSnapshot[applied={T|T < 7023417634119819264}]
I20240503 01:59:30.850651 7145 raft_consensus.cc:1261] T 789162c6428c49fb9c0edd30960d13c1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:30.851612 7276 raft_consensus.cc:1261] T 789162c6428c49fb9c0edd30960d13c1 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:30.852157 8642 consensus_queue.cc:1035] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:30.852488 8642 consensus_queue.cc:1035] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 136.903 ms
time per row: 0.0684514 ms
I20240503 01:59:30.916690 6741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:30.923417 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 79efb54e1f8143be868e9ef37f5f925d (DEFAULT_TABLE table=rebalance_test_table_13 [id=06feff591c384a50825820a47b49c8f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.923678 6863 tablet_service.cc:1467] Processing CreateTablet for tablet 8ad236ea5ab347c69026293bef7c04a8 (DEFAULT_TABLE table=rebalance_test_table_13 [id=06feff591c384a50825820a47b49c8f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.923800 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79efb54e1f8143be868e9ef37f5f925d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.923938 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ad236ea5ab347c69026293bef7c04a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.924487 7255 tablet_service.cc:1467] Processing CreateTablet for tablet 79efb54e1f8143be868e9ef37f5f925d (DEFAULT_TABLE table=rebalance_test_table_13 [id=06feff591c384a50825820a47b49c8f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.924702 7124 tablet_service.cc:1467] Processing CreateTablet for tablet 79efb54e1f8143be868e9ef37f5f925d (DEFAULT_TABLE table=rebalance_test_table_13 [id=06feff591c384a50825820a47b49c8f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.924789 7255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79efb54e1f8143be868e9ef37f5f925d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.924969 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79efb54e1f8143be868e9ef37f5f925d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.926769 8805 tablet_bootstrap.cc:492] T 79efb54e1f8143be868e9ef37f5f925d P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:30.926810 7123 tablet_service.cc:1467] Processing CreateTablet for tablet 8ad236ea5ab347c69026293bef7c04a8 (DEFAULT_TABLE table=rebalance_test_table_13 [id=06feff591c384a50825820a47b49c8f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.927064 7123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ad236ea5ab347c69026293bef7c04a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.927198 8806 tablet_bootstrap.cc:492] T 79efb54e1f8143be868e9ef37f5f925d P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:30.927903 8805 tablet_bootstrap.cc:654] T 79efb54e1f8143be868e9ef37f5f925d P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.928081 8806 tablet_bootstrap.cc:654] T 79efb54e1f8143be868e9ef37f5f925d P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.928385 6994 tablet_service.cc:1467] Processing CreateTablet for tablet 85d3e89fac21402387aa812f754cf5bc (DEFAULT_TABLE table=rebalance_test_table_13 [id=06feff591c384a50825820a47b49c8f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.928659 6994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85d3e89fac21402387aa812f754cf5bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.929239 8806 tablet_bootstrap.cc:492] T 79efb54e1f8143be868e9ef37f5f925d P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:30.929286 8809 tablet_bootstrap.cc:492] T 8ad236ea5ab347c69026293bef7c04a8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:30.929363 8806 ts_tablet_manager.cc:1367] T 79efb54e1f8143be868e9ef37f5f925d P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:30.929692 7125 tablet_service.cc:1467] Processing CreateTablet for tablet 85d3e89fac21402387aa812f754cf5bc (DEFAULT_TABLE table=rebalance_test_table_13 [id=06feff591c384a50825820a47b49c8f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.929853 8806 raft_consensus.cc:348] T 79efb54e1f8143be868e9ef37f5f925d P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:30.929946 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85d3e89fac21402387aa812f754cf5bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.929977 8806 raft_consensus.cc:374] T 79efb54e1f8143be868e9ef37f5f925d P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.930011 8806 raft_consensus.cc:729] T 79efb54e1f8143be868e9ef37f5f925d P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.930335 8809 tablet_bootstrap.cc:654] T 8ad236ea5ab347c69026293bef7c04a8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.930271 8806 consensus_queue.cc:260] T 79efb54e1f8143be868e9ef37f5f925d P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:30.930711 8806 ts_tablet_manager.cc:1398] T 79efb54e1f8143be868e9ef37f5f925d P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:30.931277 8811 tablet_bootstrap.cc:492] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:30.931449 6863 tablet_service.cc:1467] Processing CreateTablet for tablet 85d3e89fac21402387aa812f754cf5bc (DEFAULT_TABLE table=rebalance_test_table_13 [id=06feff591c384a50825820a47b49c8f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.931702 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85d3e89fac21402387aa812f754cf5bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.932150 8811 tablet_bootstrap.cc:654] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.932780 7255 tablet_service.cc:1467] Processing CreateTablet for tablet 8ad236ea5ab347c69026293bef7c04a8 (DEFAULT_TABLE table=rebalance_test_table_13 [id=06feff591c384a50825820a47b49c8f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:30.933040 8809 tablet_bootstrap.cc:492] T 8ad236ea5ab347c69026293bef7c04a8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:30.933064 7255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ad236ea5ab347c69026293bef7c04a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:30.933133 8809 ts_tablet_manager.cc:1367] T 8ad236ea5ab347c69026293bef7c04a8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:30.933626 8809 raft_consensus.cc:348] T 8ad236ea5ab347c69026293bef7c04a8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:30.933748 8809 raft_consensus.cc:374] T 8ad236ea5ab347c69026293bef7c04a8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.933779 8809 raft_consensus.cc:729] T 8ad236ea5ab347c69026293bef7c04a8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.933916 8809 consensus_queue.cc:260] T 8ad236ea5ab347c69026293bef7c04a8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:30.934860 8809 ts_tablet_manager.cc:1398] T 8ad236ea5ab347c69026293bef7c04a8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:30.936216 8806 tablet_bootstrap.cc:492] T 8ad236ea5ab347c69026293bef7c04a8 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:30.937091 8809 tablet_bootstrap.cc:492] T 85d3e89fac21402387aa812f754cf5bc P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:30.937530 8806 tablet_bootstrap.cc:654] T 8ad236ea5ab347c69026293bef7c04a8 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.938150 8809 tablet_bootstrap.cc:654] T 85d3e89fac21402387aa812f754cf5bc P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.939158 8811 tablet_bootstrap.cc:492] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:30.939270 8811 ts_tablet_manager.cc:1367] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240503 01:59:30.939301 8806 tablet_bootstrap.cc:492] T 8ad236ea5ab347c69026293bef7c04a8 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:30.939301 8809 tablet_bootstrap.cc:492] T 85d3e89fac21402387aa812f754cf5bc P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:30.939390 8809 ts_tablet_manager.cc:1367] T 85d3e89fac21402387aa812f754cf5bc P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:30.939390 8806 ts_tablet_manager.cc:1367] T 8ad236ea5ab347c69026293bef7c04a8 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:30.939751 8811 raft_consensus.cc:348] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:30.939865 8811 raft_consensus.cc:374] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.939903 8811 raft_consensus.cc:729] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.939862 8806 raft_consensus.cc:348] T 8ad236ea5ab347c69026293bef7c04a8 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:30.939862 8809 raft_consensus.cc:348] T 85d3e89fac21402387aa812f754cf5bc P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:30.939968 8806 raft_consensus.cc:374] T 8ad236ea5ab347c69026293bef7c04a8 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.939973 8809 raft_consensus.cc:374] T 85d3e89fac21402387aa812f754cf5bc P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.940022 8806 raft_consensus.cc:729] T 8ad236ea5ab347c69026293bef7c04a8 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.940040 8809 raft_consensus.cc:729] T 85d3e89fac21402387aa812f754cf5bc P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.940043 8811 consensus_queue.cc:260] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:30.940178 8809 consensus_queue.cc:260] T 85d3e89fac21402387aa812f754cf5bc P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:30.940392 8811 ts_tablet_manager.cc:1398] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:30.940518 8809 ts_tablet_manager.cc:1398] T 85d3e89fac21402387aa812f754cf5bc P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:30.940698 8809 tablet_bootstrap.cc:492] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:30.941198 8805 tablet_bootstrap.cc:492] T 79efb54e1f8143be868e9ef37f5f925d P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:30.941299 8805 ts_tablet_manager.cc:1367] T 79efb54e1f8143be868e9ef37f5f925d P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 01:59:30.941735 8809 tablet_bootstrap.cc:654] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.941794 8805 raft_consensus.cc:348] T 79efb54e1f8143be868e9ef37f5f925d P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:30.941906 8805 raft_consensus.cc:374] T 79efb54e1f8143be868e9ef37f5f925d P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.941944 8805 raft_consensus.cc:729] T 79efb54e1f8143be868e9ef37f5f925d P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.942070 8805 consensus_queue.cc:260] T 79efb54e1f8143be868e9ef37f5f925d P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:30.942124 8806 consensus_queue.cc:260] T 8ad236ea5ab347c69026293bef7c04a8 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:30.942382 8805 ts_tablet_manager.cc:1398] T 79efb54e1f8143be868e9ef37f5f925d P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:30.943359 8806 ts_tablet_manager.cc:1398] T 8ad236ea5ab347c69026293bef7c04a8 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:30.944784 8805 tablet_bootstrap.cc:492] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:30.945755 8805 tablet_bootstrap.cc:654] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.947727 8809 tablet_bootstrap.cc:492] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:30.947846 8809 ts_tablet_manager.cc:1367] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240503 01:59:30.948415 8809 raft_consensus.cc:348] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:30.948578 8809 raft_consensus.cc:374] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.948624 8809 raft_consensus.cc:729] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.948751 8809 consensus_queue.cc:260] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:30.949038 8809 ts_tablet_manager.cc:1398] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 01:59:30.952066 8805 tablet_bootstrap.cc:492] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:30.952234 8805 ts_tablet_manager.cc:1367] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 01:59:30.952718 8805 raft_consensus.cc:348] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:30.952831 8805 raft_consensus.cc:374] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.952864 8805 raft_consensus.cc:729] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.953008 8805 consensus_queue.cc:260] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:30.953254 8805 ts_tablet_manager.cc:1398] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:30.953440 8805 tablet_bootstrap.cc:492] T 85d3e89fac21402387aa812f754cf5bc P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:30.954456 8805 tablet_bootstrap.cc:654] T 85d3e89fac21402387aa812f754cf5bc P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:30.956180 8805 tablet_bootstrap.cc:492] T 85d3e89fac21402387aa812f754cf5bc P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:30.956322 8805 ts_tablet_manager.cc:1367] T 85d3e89fac21402387aa812f754cf5bc P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:30.957069 8805 raft_consensus.cc:348] T 85d3e89fac21402387aa812f754cf5bc P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:30.957235 8805 raft_consensus.cc:374] T 85d3e89fac21402387aa812f754cf5bc P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:30.957280 8805 raft_consensus.cc:729] T 85d3e89fac21402387aa812f754cf5bc P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:30.957468 8805 consensus_queue.cc:260] T 85d3e89fac21402387aa812f754cf5bc P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:30.957947 8805 ts_tablet_manager.cc:1398] T 85d3e89fac21402387aa812f754cf5bc P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:31.024293 8397 raft_consensus.cc:482] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:31.024461 8397 raft_consensus.cc:504] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:31.024910 8397 leader_election.cc:288] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:31.025396 7276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ad236ea5ab347c69026293bef7c04a8" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:31.025396 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ad236ea5ab347c69026293bef7c04a8" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:31.025547 7145 raft_consensus.cc:2454] T 8ad236ea5ab347c69026293bef7c04a8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:31.025548 7276 raft_consensus.cc:2454] T 8ad236ea5ab347c69026293bef7c04a8 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:31.025903 6817 leader_election.cc:302] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:31.026084 8397 raft_consensus.cc:2781] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:31.026144 8397 raft_consensus.cc:482] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:31.026171 8397 raft_consensus.cc:3037] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.027266 8397 raft_consensus.cc:504] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:31.027679 8397 leader_election.cc:288] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Requested vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:31.028084 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ad236ea5ab347c69026293bef7c04a8" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:31.028213 7145 raft_consensus.cc:3037] T 8ad236ea5ab347c69026293bef7c04a8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.029039 7145 raft_consensus.cc:2454] T 8ad236ea5ab347c69026293bef7c04a8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:31.029333 7276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ad236ea5ab347c69026293bef7c04a8" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:31.029381 6818 leader_election.cc:302] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:31.029465 7276 raft_consensus.cc:3037] T 8ad236ea5ab347c69026293bef7c04a8 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.029587 8397 raft_consensus.cc:2781] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:31.029784 8397 raft_consensus.cc:686] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 01:59:31.029980 8397 consensus_queue.cc:237] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:31.030550 7276 raft_consensus.cc:2454] T 8ad236ea5ab347c69026293bef7c04a8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:31.032017 6741 catalog_manager.cc:5484] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:31.049849 8787 raft_consensus.cc:482] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:31.050010 8787 raft_consensus.cc:504] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:31.050455 8787 leader_election.cc:288] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:31.050967 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d3e89fac21402387aa812f754cf5bc" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:31.050969 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d3e89fac21402387aa812f754cf5bc" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:31.051117 7145 raft_consensus.cc:2454] T 85d3e89fac21402387aa812f754cf5bc P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:31.051121 6882 raft_consensus.cc:2454] T 85d3e89fac21402387aa812f754cf5bc P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:31.051448 6949 leader_election.cc:302] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:31.051637 8787 raft_consensus.cc:2781] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:31.051695 8787 raft_consensus.cc:482] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:31.051730 8787 raft_consensus.cc:3037] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.052637 8787 raft_consensus.cc:504] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:31.052994 8787 leader_election.cc:288] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:31.053357 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d3e89fac21402387aa812f754cf5bc" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:31.053368 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d3e89fac21402387aa812f754cf5bc" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:31.053465 6882 raft_consensus.cc:3037] T 85d3e89fac21402387aa812f754cf5bc P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.053521 7145 raft_consensus.cc:3037] T 85d3e89fac21402387aa812f754cf5bc P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.054389 7145 raft_consensus.cc:2454] T 85d3e89fac21402387aa812f754cf5bc P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:31.054397 6882 raft_consensus.cc:2454] T 85d3e89fac21402387aa812f754cf5bc P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:31.054745 6950 leader_election.cc:302] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:31.054960 8787 raft_consensus.cc:2781] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:31.055078 8787 raft_consensus.cc:686] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:31.055294 8787 consensus_queue.cc:237] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:31.057765 6741 catalog_manager.cc:5484] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:31.173677 8648 raft_consensus.cc:482] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:31.173836 8648 raft_consensus.cc:504] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:31.174371 8648 leader_election.cc:288] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:31.174878 7276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79efb54e1f8143be868e9ef37f5f925d" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:31.174878 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79efb54e1f8143be868e9ef37f5f925d" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:31.175012 7276 raft_consensus.cc:2454] T 79efb54e1f8143be868e9ef37f5f925d P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:31.175034 6882 raft_consensus.cc:2454] T 79efb54e1f8143be868e9ef37f5f925d P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:31.175347 7081 leader_election.cc:302] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:31.175520 8648 raft_consensus.cc:2781] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:31.175580 8648 raft_consensus.cc:482] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:31.175624 8648 raft_consensus.cc:3037] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.176474 8648 raft_consensus.cc:504] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:31.176836 8648 leader_election.cc:288] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Requested vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:31.177338 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79efb54e1f8143be868e9ef37f5f925d" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:31.177475 7276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79efb54e1f8143be868e9ef37f5f925d" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:31.177584 6882 raft_consensus.cc:3037] T 79efb54e1f8143be868e9ef37f5f925d P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.177628 7276 raft_consensus.cc:3037] T 79efb54e1f8143be868e9ef37f5f925d P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.179093 6882 raft_consensus.cc:2454] T 79efb54e1f8143be868e9ef37f5f925d P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:31.179090 7276 raft_consensus.cc:2454] T 79efb54e1f8143be868e9ef37f5f925d P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:31.179461 7081 leader_election.cc:302] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:31.179797 8648 raft_consensus.cc:2781] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:31.179888 8648 raft_consensus.cc:686] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Becoming Leader. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Running, Role: LEADER
I20240503 01:59:31.180045 8648 consensus_queue.cc:237] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:31.182255 6741 catalog_manager.cc:5484] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
W20240503 01:59:31.354663 8822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:31.354982 8822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:31.358783 8822 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_13'
I20240503 01:59:31.471679 6883 raft_consensus.cc:1261] T 79efb54e1f8143be868e9ef37f5f925d P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:31.471760 7276 raft_consensus.cc:1261] T 79efb54e1f8143be868e9ef37f5f925d P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:31.472230 8648 consensus_queue.cc:1035] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:31.472389 8647 consensus_queue.cc:1035] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:31.473744 7145 raft_consensus.cc:1261] T 85d3e89fac21402387aa812f754cf5bc P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:31.474184 8787 consensus_queue.cc:1035] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:31.476997 7275 raft_consensus.cc:1261] T 8ad236ea5ab347c69026293bef7c04a8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:31.477411 7143 raft_consensus.cc:1261] T 8ad236ea5ab347c69026293bef7c04a8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:31.477849 8397 consensus_queue.cc:1035] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:31.478188 8818 consensus_queue.cc:1035] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:31.494155 6882 raft_consensus.cc:1261] T 85d3e89fac21402387aa812f754cf5bc P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:31.494891 8742 consensus_queue.cc:1035] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:31.503742 8865 mvcc.cc:205] Tried to move back new op lower bound from 7023417636741283840 to 7023417635044098048. Current Snapshot: MvccSnapshot[applied={T|T < 7023417636741283840}]
INSERT report
rows total: 2000
time total: 151.651 ms
time per row: 0.0758254 ms
I20240503 01:59:31.565248 6733 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:31.572497 6994 tablet_service.cc:1467] Processing CreateTablet for tablet 580e6b4d2b2f4f79b8c2af5e75a3c75f (DEFAULT_TABLE table=rebalance_test_table_14 [id=c0195ea81aa24df495aff19543a2b8eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.572862 6994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 580e6b4d2b2f4f79b8c2af5e75a3c75f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.573294 6863 tablet_service.cc:1467] Processing CreateTablet for tablet 580e6b4d2b2f4f79b8c2af5e75a3c75f (DEFAULT_TABLE table=rebalance_test_table_14 [id=c0195ea81aa24df495aff19543a2b8eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.573689 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 580e6b4d2b2f4f79b8c2af5e75a3c75f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.573829 7124 tablet_service.cc:1467] Processing CreateTablet for tablet f8f3bfca68014d818a58484d248a9846 (DEFAULT_TABLE table=rebalance_test_table_14 [id=c0195ea81aa24df495aff19543a2b8eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.574131 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8f3bfca68014d818a58484d248a9846. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.575006 8885 tablet_bootstrap.cc:492] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:31.575469 6993 tablet_service.cc:1467] Processing CreateTablet for tablet c35ecf8890b64caaac7e5ea9f6e8f989 (DEFAULT_TABLE table=rebalance_test_table_14 [id=c0195ea81aa24df495aff19543a2b8eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.575731 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35ecf8890b64caaac7e5ea9f6e8f989. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.575990 8885 tablet_bootstrap.cc:654] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.576694 8888 tablet_bootstrap.cc:492] T f8f3bfca68014d818a58484d248a9846 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:31.573294 6861 tablet_service.cc:1467] Processing CreateTablet for tablet c35ecf8890b64caaac7e5ea9f6e8f989 (DEFAULT_TABLE table=rebalance_test_table_14 [id=c0195ea81aa24df495aff19543a2b8eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.576879 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35ecf8890b64caaac7e5ea9f6e8f989. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.577807 8888 tablet_bootstrap.cc:654] T f8f3bfca68014d818a58484d248a9846 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.578752 6994 tablet_service.cc:1467] Processing CreateTablet for tablet f8f3bfca68014d818a58484d248a9846 (DEFAULT_TABLE table=rebalance_test_table_14 [id=c0195ea81aa24df495aff19543a2b8eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.579028 6994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8f3bfca68014d818a58484d248a9846. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.580040 7255 tablet_service.cc:1467] Processing CreateTablet for tablet 580e6b4d2b2f4f79b8c2af5e75a3c75f (DEFAULT_TABLE table=rebalance_test_table_14 [id=c0195ea81aa24df495aff19543a2b8eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.580086 7254 tablet_service.cc:1467] Processing CreateTablet for tablet f8f3bfca68014d818a58484d248a9846 (DEFAULT_TABLE table=rebalance_test_table_14 [id=c0195ea81aa24df495aff19543a2b8eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.580346 7255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 580e6b4d2b2f4f79b8c2af5e75a3c75f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.580086 7256 tablet_service.cc:1467] Processing CreateTablet for tablet c35ecf8890b64caaac7e5ea9f6e8f989 (DEFAULT_TABLE table=rebalance_test_table_14 [id=c0195ea81aa24df495aff19543a2b8eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:31.580592 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35ecf8890b64caaac7e5ea9f6e8f989. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.580592 8888 tablet_bootstrap.cc:492] T f8f3bfca68014d818a58484d248a9846 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:31.580687 8888 ts_tablet_manager.cc:1367] T f8f3bfca68014d818a58484d248a9846 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 01:59:31.581197 8888 raft_consensus.cc:348] T f8f3bfca68014d818a58484d248a9846 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:31.581310 8888 raft_consensus.cc:374] T f8f3bfca68014d818a58484d248a9846 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.581350 8888 raft_consensus.cc:729] T f8f3bfca68014d818a58484d248a9846 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.581494 8888 consensus_queue.cc:260] T f8f3bfca68014d818a58484d248a9846 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:31.581851 8888 ts_tablet_manager.cc:1398] T f8f3bfca68014d818a58484d248a9846 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:31.582396 8886 tablet_bootstrap.cc:492] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:31.582955 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8f3bfca68014d818a58484d248a9846. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:31.583756 8886 tablet_bootstrap.cc:654] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.583853 8885 tablet_bootstrap.cc:492] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:31.583976 8885 ts_tablet_manager.cc:1367] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 01:59:31.584543 8885 raft_consensus.cc:348] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:31.584707 8885 raft_consensus.cc:374] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.584764 8885 raft_consensus.cc:729] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.584923 8885 consensus_queue.cc:260] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:31.585016 8886 tablet_bootstrap.cc:492] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:31.585111 8886 ts_tablet_manager.cc:1367] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:31.585243 8885 ts_tablet_manager.cc:1398] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:31.585444 8885 tablet_bootstrap.cc:492] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:31.585635 8886 raft_consensus.cc:348] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:31.585767 8886 raft_consensus.cc:374] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.585798 8886 raft_consensus.cc:729] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.585927 8886 consensus_queue.cc:260] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:31.586181 8890 tablet_bootstrap.cc:492] T c35ecf8890b64caaac7e5ea9f6e8f989 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:31.586236 8886 ts_tablet_manager.cc:1398] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:31.586401 8886 tablet_bootstrap.cc:492] T c35ecf8890b64caaac7e5ea9f6e8f989 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:31.586587 8885 tablet_bootstrap.cc:654] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.587265 8890 tablet_bootstrap.cc:654] T c35ecf8890b64caaac7e5ea9f6e8f989 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.587347 8886 tablet_bootstrap.cc:654] T c35ecf8890b64caaac7e5ea9f6e8f989 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.589149 8885 tablet_bootstrap.cc:492] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:31.589239 8885 ts_tablet_manager.cc:1367] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 01:59:31.589704 8885 raft_consensus.cc:348] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:31.589865 8885 raft_consensus.cc:374] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.589924 8885 raft_consensus.cc:729] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.590062 8885 consensus_queue.cc:260] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:31.590363 8885 ts_tablet_manager.cc:1398] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:31.590566 8885 tablet_bootstrap.cc:492] T f8f3bfca68014d818a58484d248a9846 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:31.591616 8885 tablet_bootstrap.cc:654] T f8f3bfca68014d818a58484d248a9846 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.592178 8886 tablet_bootstrap.cc:492] T c35ecf8890b64caaac7e5ea9f6e8f989 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:31.592276 8886 ts_tablet_manager.cc:1367] T c35ecf8890b64caaac7e5ea9f6e8f989 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:31.592713 8886 raft_consensus.cc:348] T c35ecf8890b64caaac7e5ea9f6e8f989 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:31.592797 8885 tablet_bootstrap.cc:492] T f8f3bfca68014d818a58484d248a9846 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:31.592825 8886 raft_consensus.cc:374] T c35ecf8890b64caaac7e5ea9f6e8f989 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.592861 8886 raft_consensus.cc:729] T c35ecf8890b64caaac7e5ea9f6e8f989 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.592895 8885 ts_tablet_manager.cc:1367] T f8f3bfca68014d818a58484d248a9846 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 01:59:31.592995 8886 consensus_queue.cc:260] T c35ecf8890b64caaac7e5ea9f6e8f989 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:31.593266 8886 ts_tablet_manager.cc:1398] T c35ecf8890b64caaac7e5ea9f6e8f989 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:31.593406 8885 raft_consensus.cc:348] T f8f3bfca68014d818a58484d248a9846 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:31.593515 8885 raft_consensus.cc:374] T f8f3bfca68014d818a58484d248a9846 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.593554 8885 raft_consensus.cc:729] T f8f3bfca68014d818a58484d248a9846 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.593667 8885 consensus_queue.cc:260] T f8f3bfca68014d818a58484d248a9846 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:31.593933 8885 ts_tablet_manager.cc:1398] T f8f3bfca68014d818a58484d248a9846 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:31.595665 8890 tablet_bootstrap.cc:492] T c35ecf8890b64caaac7e5ea9f6e8f989 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:31.595943 8890 ts_tablet_manager.cc:1367] T c35ecf8890b64caaac7e5ea9f6e8f989 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 01:59:31.596446 8890 raft_consensus.cc:348] T c35ecf8890b64caaac7e5ea9f6e8f989 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:31.596555 8890 raft_consensus.cc:374] T c35ecf8890b64caaac7e5ea9f6e8f989 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.596588 8890 raft_consensus.cc:729] T c35ecf8890b64caaac7e5ea9f6e8f989 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.596729 8890 consensus_queue.cc:260] T c35ecf8890b64caaac7e5ea9f6e8f989 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:31.597019 8890 ts_tablet_manager.cc:1398] T c35ecf8890b64caaac7e5ea9f6e8f989 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:31.597205 8890 tablet_bootstrap.cc:492] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:31.598269 8890 tablet_bootstrap.cc:654] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.599726 8890 tablet_bootstrap.cc:492] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:31.599843 8890 ts_tablet_manager.cc:1367] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:31.600425 8890 raft_consensus.cc:348] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:31.600538 8890 raft_consensus.cc:374] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.600572 8890 raft_consensus.cc:729] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.600708 8890 consensus_queue.cc:260] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:31.600993 8890 ts_tablet_manager.cc:1398] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:31.601169 8890 tablet_bootstrap.cc:492] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:31.602136 8890 tablet_bootstrap.cc:654] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:31.603343 8890 tablet_bootstrap.cc:492] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:31.603451 8890 ts_tablet_manager.cc:1367] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:31.603951 8890 raft_consensus.cc:348] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:31.604058 8890 raft_consensus.cc:374] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:31.604079 8890 raft_consensus.cc:729] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:31.604190 8890 consensus_queue.cc:260] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:31.604480 8890 ts_tablet_manager.cc:1398] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:31.642083 8079 raft_consensus.cc:482] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:31.642369 8079 raft_consensus.cc:504] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:31.642962 8079 leader_election.cc:288] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:31.643563 7013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "580e6b4d2b2f4f79b8c2af5e75a3c75f" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:31.643551 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "580e6b4d2b2f4f79b8c2af5e75a3c75f" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:31.643720 7013 raft_consensus.cc:2454] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:31.643751 6882 raft_consensus.cc:2454] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:31.644083 7212 leader_election.cc:302] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:31.644308 8720 raft_consensus.cc:2781] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:31.644372 8720 raft_consensus.cc:482] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:31.644450 8720 raft_consensus.cc:3037] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.645392 8720 raft_consensus.cc:504] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:31.645797 8720 leader_election.cc:288] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Requested vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:31.646317 7013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "580e6b4d2b2f4f79b8c2af5e75a3c75f" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:31.646546 7013 raft_consensus.cc:3037] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.646476 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "580e6b4d2b2f4f79b8c2af5e75a3c75f" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:31.646665 6882 raft_consensus.cc:3037] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.647765 6882 raft_consensus.cc:2454] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:31.647765 7013 raft_consensus.cc:2454] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:31.648149 7212 leader_election.cc:302] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:31.648332 8720 raft_consensus.cc:2781] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:31.648411 8720 raft_consensus.cc:686] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 01:59:31.648578 8720 consensus_queue.cc:237] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:31.651062 6738 catalog_manager.cc:5484] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:31.689882 8720 raft_consensus.cc:482] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:31.690032 8720 raft_consensus.cc:504] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:31.690567 8720 leader_election.cc:288] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:31.691067 7013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f3bfca68014d818a58484d248a9846" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:31.691076 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f3bfca68014d818a58484d248a9846" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:31.691219 7145 raft_consensus.cc:2454] T f8f3bfca68014d818a58484d248a9846 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:31.691251 7013 raft_consensus.cc:2454] T f8f3bfca68014d818a58484d248a9846 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:31.691572 7211 leader_election.cc:302] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:31.691793 8720 raft_consensus.cc:2781] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:31.691850 8720 raft_consensus.cc:482] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:31.691876 8720 raft_consensus.cc:3037] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.692747 8720 raft_consensus.cc:504] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:31.693156 8720 leader_election.cc:288] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Requested vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:31.693756 7013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f3bfca68014d818a58484d248a9846" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:31.693760 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f3bfca68014d818a58484d248a9846" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:31.693890 7145 raft_consensus.cc:3037] T f8f3bfca68014d818a58484d248a9846 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.693905 7013 raft_consensus.cc:3037] T f8f3bfca68014d818a58484d248a9846 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.694846 7013 raft_consensus.cc:2454] T f8f3bfca68014d818a58484d248a9846 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:31.694856 7145 raft_consensus.cc:2454] T f8f3bfca68014d818a58484d248a9846 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:31.695238 7209 leader_election.cc:302] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:31.695467 8720 raft_consensus.cc:2781] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:31.695580 8720 raft_consensus.cc:686] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 01:59:31.695737 8720 consensus_queue.cc:237] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:31.698138 6738 catalog_manager.cc:5484] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:31.729249 8742 raft_consensus.cc:482] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:31.729532 8742 raft_consensus.cc:504] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:31.730470 8742 leader_election.cc:288] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:31.730960 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35ecf8890b64caaac7e5ea9f6e8f989" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:31.731164 7275 raft_consensus.cc:2454] T c35ecf8890b64caaac7e5ea9f6e8f989 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:31.731164 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35ecf8890b64caaac7e5ea9f6e8f989" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:31.731395 6882 raft_consensus.cc:2454] T c35ecf8890b64caaac7e5ea9f6e8f989 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:31.731660 6948 leader_election.cc:302] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:31.731879 8742 raft_consensus.cc:2781] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:31.731937 8742 raft_consensus.cc:482] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:31.731964 8742 raft_consensus.cc:3037] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.732858 8742 raft_consensus.cc:504] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:31.733250 8742 leader_election.cc:288] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:31.733847 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35ecf8890b64caaac7e5ea9f6e8f989" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:31.733842 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35ecf8890b64caaac7e5ea9f6e8f989" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:31.733974 6882 raft_consensus.cc:3037] T c35ecf8890b64caaac7e5ea9f6e8f989 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.733995 7275 raft_consensus.cc:3037] T c35ecf8890b64caaac7e5ea9f6e8f989 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:31.735205 7275 raft_consensus.cc:2454] T c35ecf8890b64caaac7e5ea9f6e8f989 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:31.735205 6882 raft_consensus.cc:2454] T c35ecf8890b64caaac7e5ea9f6e8f989 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:31.735684 6950 leader_election.cc:302] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:31.736081 8742 raft_consensus.cc:2781] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:31.736192 8742 raft_consensus.cc:686] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:31.736359 8742 consensus_queue.cc:237] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:31.739009 6738 catalog_manager.cc:5484] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
W20240503 01:59:31.905313 8901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:31.905707 8901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:31.910562 8901 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_14'
I20240503 01:59:32.028108 6883 raft_consensus.cc:1261] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.028482 7013 raft_consensus.cc:1261] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.028717 8720 consensus_queue.cc:1035] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.028872 8079 consensus_queue.cc:1035] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.029613 7143 raft_consensus.cc:1261] T f8f3bfca68014d818a58484d248a9846 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.030043 8079 consensus_queue.cc:1035] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.034799 6882 raft_consensus.cc:1261] T c35ecf8890b64caaac7e5ea9f6e8f989 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.035799 8873 consensus_queue.cc:1035] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.036412 7275 raft_consensus.cc:1261] T c35ecf8890b64caaac7e5ea9f6e8f989 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.037353 7014 raft_consensus.cc:1261] T f8f3bfca68014d818a58484d248a9846 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.038362 8873 consensus_queue.cc:1035] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.038774 8720 consensus_queue.cc:1035] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.056681 8934 mvcc.cc:205] Tried to move back new op lower bound from 7023417639028416512 to 7023417637667672064. Current Snapshot: MvccSnapshot[applied={T|T < 7023417639028416512}]
I20240503 01:59:32.067915 8857 mvcc.cc:205] Tried to move back new op lower bound from 7023417639054073856 to 7023417637834633216. Current Snapshot: MvccSnapshot[applied={T|T < 7023417639038009344}]
INSERT report
rows total: 2000
time total: 144.787 ms
time per row: 0.0723933 ms
I20240503 01:59:32.113993 6736 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:32.122604 6861 tablet_service.cc:1467] Processing CreateTablet for tablet db656aaa7b104b3182350de320f67839 (DEFAULT_TABLE table=rebalance_test_table_15 [id=76d0e687959345af9b8f30fe2c35d640]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.122723 6993 tablet_service.cc:1467] Processing CreateTablet for tablet c6833c997d284063955d0b9b09ccb3b7 (DEFAULT_TABLE table=rebalance_test_table_15 [id=76d0e687959345af9b8f30fe2c35d640]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.122723 6994 tablet_service.cc:1467] Processing CreateTablet for tablet db656aaa7b104b3182350de320f67839 (DEFAULT_TABLE table=rebalance_test_table_15 [id=76d0e687959345af9b8f30fe2c35d640]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.122964 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db656aaa7b104b3182350de320f67839. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.123085 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6833c997d284063955d0b9b09ccb3b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.123252 6994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db656aaa7b104b3182350de320f67839. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.125398 8952 tablet_bootstrap.cc:492] T db656aaa7b104b3182350de320f67839 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:32.125701 7124 tablet_service.cc:1467] Processing CreateTablet for tablet db656aaa7b104b3182350de320f67839 (DEFAULT_TABLE table=rebalance_test_table_15 [id=76d0e687959345af9b8f30fe2c35d640]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.125993 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db656aaa7b104b3182350de320f67839. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.126435 8953 tablet_bootstrap.cc:492] T db656aaa7b104b3182350de320f67839 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:32.126791 8952 tablet_bootstrap.cc:654] T db656aaa7b104b3182350de320f67839 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.127439 8953 tablet_bootstrap.cc:654] T db656aaa7b104b3182350de320f67839 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.128480 8956 tablet_bootstrap.cc:492] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:32.128818 6863 tablet_service.cc:1467] Processing CreateTablet for tablet 9c7396d1b98c4b7ca495ba3ab5dcbbc6 (DEFAULT_TABLE table=rebalance_test_table_15 [id=76d0e687959345af9b8f30fe2c35d640]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.129065 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7396d1b98c4b7ca495ba3ab5dcbbc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.129150 8953 tablet_bootstrap.cc:492] T db656aaa7b104b3182350de320f67839 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:32.129238 8953 ts_tablet_manager.cc:1367] T db656aaa7b104b3182350de320f67839 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:32.129485 8956 tablet_bootstrap.cc:654] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.129746 8953 raft_consensus.cc:348] T db656aaa7b104b3182350de320f67839 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.129859 8953 raft_consensus.cc:374] T db656aaa7b104b3182350de320f67839 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.129900 8953 raft_consensus.cc:729] T db656aaa7b104b3182350de320f67839 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.130095 8953 consensus_queue.cc:260] T db656aaa7b104b3182350de320f67839 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.130393 8953 ts_tablet_manager.cc:1398] T db656aaa7b104b3182350de320f67839 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.130657 8953 tablet_bootstrap.cc:492] T c6833c997d284063955d0b9b09ccb3b7 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:32.130738 8956 tablet_bootstrap.cc:492] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:32.130815 8956 ts_tablet_manager.cc:1367] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:32.131388 7125 tablet_service.cc:1467] Processing CreateTablet for tablet c6833c997d284063955d0b9b09ccb3b7 (DEFAULT_TABLE table=rebalance_test_table_15 [id=76d0e687959345af9b8f30fe2c35d640]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.131613 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6833c997d284063955d0b9b09ccb3b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.132054 8956 raft_consensus.cc:348] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.132483 8956 raft_consensus.cc:374] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.132530 8956 raft_consensus.cc:729] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.132675 8956 consensus_queue.cc:260] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.133455 8953 tablet_bootstrap.cc:654] T c6833c997d284063955d0b9b09ccb3b7 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.133821 7123 tablet_service.cc:1467] Processing CreateTablet for tablet 9c7396d1b98c4b7ca495ba3ab5dcbbc6 (DEFAULT_TABLE table=rebalance_test_table_15 [id=76d0e687959345af9b8f30fe2c35d640]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.134014 7123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7396d1b98c4b7ca495ba3ab5dcbbc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.133993 8956 ts_tablet_manager.cc:1398] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:32.134168 8956 tablet_bootstrap.cc:492] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:32.135172 8956 tablet_bootstrap.cc:654] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.136588 8953 tablet_bootstrap.cc:492] T c6833c997d284063955d0b9b09ccb3b7 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:32.136648 8647 raft_consensus.cc:482] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:32.136757 8953 ts_tablet_manager.cc:1367] T c6833c997d284063955d0b9b09ccb3b7 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 01:59:32.136745 8647 raft_consensus.cc:504] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.136952 8956 tablet_bootstrap.cc:492] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:32.137045 8956 ts_tablet_manager.cc:1367] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:32.137168 8647 leader_election.cc:288] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:32.137462 8953 raft_consensus.cc:348] T c6833c997d284063955d0b9b09ccb3b7 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:32.137586 8953 raft_consensus.cc:374] T c6833c997d284063955d0b9b09ccb3b7 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.137538 8956 raft_consensus.cc:348] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:32.137626 8953 raft_consensus.cc:729] T c6833c997d284063955d0b9b09ccb3b7 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.137658 8956 raft_consensus.cc:374] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.137692 8956 raft_consensus.cc:729] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.137758 8953 consensus_queue.cc:260] T c6833c997d284063955d0b9b09ccb3b7 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:32.137810 8956 consensus_queue.cc:260] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:32.138093 8956 ts_tablet_manager.cc:1398] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.138057 8953 ts_tablet_manager.cc:1398] T c6833c997d284063955d0b9b09ccb3b7 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:32.138486 7013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db656aaa7b104b3182350de320f67839" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:32.138424 7255 tablet_service.cc:1467] Processing CreateTablet for tablet c6833c997d284063955d0b9b09ccb3b7 (DEFAULT_TABLE table=rebalance_test_table_15 [id=76d0e687959345af9b8f30fe2c35d640]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.138638 7013 raft_consensus.cc:2454] T db656aaa7b104b3182350de320f67839 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:32.138885 7255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6833c997d284063955d0b9b09ccb3b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.138985 7078 leader_election.cc:302] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:32.139163 8647 raft_consensus.cc:2781] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:32.139214 8647 raft_consensus.cc:482] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:32.139248 8647 raft_consensus.cc:3037] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.140362 8647 raft_consensus.cc:504] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.140751 8647 leader_election.cc:288] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Requested vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:32.141258 7013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db656aaa7b104b3182350de320f67839" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:32.141387 7013 raft_consensus.cc:3037] T db656aaa7b104b3182350de320f67839 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.141914 8960 tablet_bootstrap.cc:492] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:32.142213 7254 tablet_service.cc:1467] Processing CreateTablet for tablet 9c7396d1b98c4b7ca495ba3ab5dcbbc6 (DEFAULT_TABLE table=rebalance_test_table_15 [id=76d0e687959345af9b8f30fe2c35d640]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.142470 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7396d1b98c4b7ca495ba3ab5dcbbc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.142546 7013 raft_consensus.cc:2454] T db656aaa7b104b3182350de320f67839 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:32.142884 7078 leader_election.cc:302] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:32.143131 8960 tablet_bootstrap.cc:654] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.143188 8647 raft_consensus.cc:2781] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:32.143277 8647 raft_consensus.cc:686] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Becoming Leader. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Running, Role: LEADER
I20240503 01:59:32.143424 8647 consensus_queue.cc:237] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.144291 8960 tablet_bootstrap.cc:492] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:32.144414 8960 ts_tablet_manager.cc:1367] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:32.144855 8960 raft_consensus.cc:348] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:32.145011 8960 raft_consensus.cc:374] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.145066 8960 raft_consensus.cc:729] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.145118 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db656aaa7b104b3182350de320f67839" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:32.145200 8960 consensus_queue.cc:260] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:32.145531 8960 ts_tablet_manager.cc:1398] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.145782 8960 tablet_bootstrap.cc:492] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:32.146828 8960 tablet_bootstrap.cc:654] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.146462 6736 catalog_manager.cc:5484] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
W20240503 01:59:32.148702 7081 leader_election.cc:341] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): Illegal state: must be running to vote when last-logged opid is not known
I20240503 01:59:32.148845 8960 tablet_bootstrap.cc:492] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:32.148953 8960 ts_tablet_manager.cc:1367] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:32.149478 8960 raft_consensus.cc:348] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.149524 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db656aaa7b104b3182350de320f67839" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:32.149648 8960 raft_consensus.cc:374] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.149729 8960 raft_consensus.cc:729] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
W20240503 01:59:32.149917 7081 leader_election.cc:341] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): Illegal state: must be running to vote when last-logged opid is not known
I20240503 01:59:32.149964 8960 consensus_queue.cc:260] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.150269 8960 ts_tablet_manager.cc:1398] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.151415 8952 tablet_bootstrap.cc:492] T db656aaa7b104b3182350de320f67839 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:32.151530 8952 ts_tablet_manager.cc:1367] T db656aaa7b104b3182350de320f67839 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240503 01:59:32.152094 8952 raft_consensus.cc:348] T db656aaa7b104b3182350de320f67839 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.152179 8956 tablet_bootstrap.cc:492] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:32.152220 8952 raft_consensus.cc:374] T db656aaa7b104b3182350de320f67839 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.152259 8952 raft_consensus.cc:729] T db656aaa7b104b3182350de320f67839 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.152372 8952 consensus_queue.cc:260] T db656aaa7b104b3182350de320f67839 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.152671 8952 ts_tablet_manager.cc:1398] T db656aaa7b104b3182350de320f67839 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.152860 8952 tablet_bootstrap.cc:492] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:32.152822 8933 raft_consensus.cc:482] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:32.152922 8933 raft_consensus.cc:504] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:32.153232 8956 tablet_bootstrap.cc:654] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.153308 8933 leader_election.cc:288] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:32.153915 8952 tablet_bootstrap.cc:654] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.153882 7013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6833c997d284063955d0b9b09ccb3b7" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:32.154019 7013 raft_consensus.cc:2454] T c6833c997d284063955d0b9b09ccb3b7 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:32.154409 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6833c997d284063955d0b9b09ccb3b7" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:32.154574 7143 raft_consensus.cc:2454] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:32.154881 7211 leader_election.cc:302] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:32.155064 8933 raft_consensus.cc:2781] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:32.155109 8933 raft_consensus.cc:482] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:32.155155 8933 raft_consensus.cc:3037] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.156138 8933 raft_consensus.cc:504] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:32.156565 8933 leader_election.cc:288] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Requested vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:32.156873 8952 tablet_bootstrap.cc:492] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:32.156823 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6833c997d284063955d0b9b09ccb3b7" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:32.156952 7143 raft_consensus.cc:3037] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.156966 8952 ts_tablet_manager.cc:1367] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:32.156971 7013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6833c997d284063955d0b9b09ccb3b7" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:32.157096 7013 raft_consensus.cc:3037] T c6833c997d284063955d0b9b09ccb3b7 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.157660 8952 raft_consensus.cc:348] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.157850 8952 raft_consensus.cc:374] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.157912 8952 raft_consensus.cc:729] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.158092 7143 raft_consensus.cc:2454] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:32.158075 8952 consensus_queue.cc:260] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.158143 7013 raft_consensus.cc:2454] T c6833c997d284063955d0b9b09ccb3b7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:32.158491 8952 ts_tablet_manager.cc:1398] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:32.158551 7209 leader_election.cc:302] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:32.158851 8933 raft_consensus.cc:2781] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:32.158943 8933 raft_consensus.cc:686] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 01:59:32.159085 8933 consensus_queue.cc:237] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:32.160372 8956 tablet_bootstrap.cc:492] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:32.160501 8956 ts_tablet_manager.cc:1367] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 01:59:32.160939 8956 raft_consensus.cc:348] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.161068 8956 raft_consensus.cc:374] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.161106 8956 raft_consensus.cc:729] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.161249 8956 consensus_queue.cc:260] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.162037 8956 ts_tablet_manager.cc:1398] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.162284 6736 catalog_manager.cc:5484] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:32.207373 8720 raft_consensus.cc:482] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:32.207520 8720 raft_consensus.cc:504] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.207978 8720 leader_election.cc:288] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:32.208549 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7396d1b98c4b7ca495ba3ab5dcbbc6" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:32.208549 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7396d1b98c4b7ca495ba3ab5dcbbc6" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:32.208698 7143 raft_consensus.cc:2454] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:32.208703 6882 raft_consensus.cc:2454] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:32.209055 7212 leader_election.cc:302] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:32.209293 8720 raft_consensus.cc:2781] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:32.209353 8720 raft_consensus.cc:482] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:32.209410 8720 raft_consensus.cc:3037] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.210559 8720 raft_consensus.cc:504] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.210965 8720 leader_election.cc:288] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Requested vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:32.211350 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7396d1b98c4b7ca495ba3ab5dcbbc6" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:32.211349 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7396d1b98c4b7ca495ba3ab5dcbbc6" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:32.211491 7143 raft_consensus.cc:3037] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.211501 6882 raft_consensus.cc:3037] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.212268 7143 raft_consensus.cc:2454] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:32.212303 6882 raft_consensus.cc:2454] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:32.212568 7211 leader_election.cc:302] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:32.212779 8720 raft_consensus.cc:2781] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:32.212867 8720 raft_consensus.cc:686] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 01:59:32.213003 8720 consensus_queue.cc:237] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.215862 6734 catalog_manager.cc:5484] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:32.219427 8397 raft_consensus.cc:482] T db656aaa7b104b3182350de320f67839 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:32.219622 8397 raft_consensus.cc:504] T db656aaa7b104b3182350de320f67839 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.220230 8397 leader_election.cc:288] T db656aaa7b104b3182350de320f67839 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:32.220937 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db656aaa7b104b3182350de320f67839" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:32.223129 7013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db656aaa7b104b3182350de320f67839" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:32.223373 7013 raft_consensus.cc:2379] T db656aaa7b104b3182350de320f67839 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5dcc33cc621a4cbbae155f3230fb7c46 in current term 1: Already voted for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in this term.
I20240503 01:59:32.223816 6816 leader_election.cc:302] T db656aaa7b104b3182350de320f67839 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a
I20240503 01:59:32.224011 8397 raft_consensus.cc:3037] T db656aaa7b104b3182350de320f67839 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.225283 8397 raft_consensus.cc:2726] T db656aaa7b104b3182350de320f67839 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 01:59:32.348758 8968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:32.349103 8968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:32.352690 8968 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_15'
I20240503 01:59:32.451043 7013 raft_consensus.cc:1261] T c6833c997d284063955d0b9b09ccb3b7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.451115 7143 raft_consensus.cc:1261] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.451756 6882 raft_consensus.cc:1261] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.451781 8079 consensus_queue.cc:1035] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.452359 8079 consensus_queue.cc:1035] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.452630 8933 consensus_queue.cc:1035] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.453243 7143 raft_consensus.cc:1261] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.462088 6882 raft_consensus.cc:1261] T db656aaa7b104b3182350de320f67839 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.462062 8079 consensus_queue.cc:1035] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.463241 8648 consensus_queue.cc:1035] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:32.472213 7013 raft_consensus.cc:1261] T db656aaa7b104b3182350de320f67839 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:32.472741 8647 consensus_queue.cc:1035] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 128.233 ms
time per row: 0.0641167 ms
I20240503 01:59:32.535022 6738 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:32.543345 6993 tablet_service.cc:1467] Processing CreateTablet for tablet b14a9a0bd39843cf8132e067fc48c29e (DEFAULT_TABLE table=rebalance_test_table_16 [id=f40118346c7d461fb1d445e8dab8b379]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.543761 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b14a9a0bd39843cf8132e067fc48c29e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.543675 6994 tablet_service.cc:1467] Processing CreateTablet for tablet 96c8dabb170f44fd98bd467f62255713 (DEFAULT_TABLE table=rebalance_test_table_16 [id=f40118346c7d461fb1d445e8dab8b379]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.543675 7255 tablet_service.cc:1467] Processing CreateTablet for tablet 96c8dabb170f44fd98bd467f62255713 (DEFAULT_TABLE table=rebalance_test_table_16 [id=f40118346c7d461fb1d445e8dab8b379]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.544016 6994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96c8dabb170f44fd98bd467f62255713. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.544018 7255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96c8dabb170f44fd98bd467f62255713. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.545130 7125 tablet_service.cc:1467] Processing CreateTablet for tablet 424123bb8be2460b8855b5a9ad785e57 (DEFAULT_TABLE table=rebalance_test_table_16 [id=f40118346c7d461fb1d445e8dab8b379]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.545452 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 424123bb8be2460b8855b5a9ad785e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.546344 8960 tablet_bootstrap.cc:492] T 96c8dabb170f44fd98bd467f62255713 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:32.546548 6863 tablet_service.cc:1467] Processing CreateTablet for tablet b14a9a0bd39843cf8132e067fc48c29e (DEFAULT_TABLE table=rebalance_test_table_16 [id=f40118346c7d461fb1d445e8dab8b379]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.546819 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b14a9a0bd39843cf8132e067fc48c29e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.547411 8960 tablet_bootstrap.cc:654] T 96c8dabb170f44fd98bd467f62255713 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.547441 7123 tablet_service.cc:1467] Processing CreateTablet for tablet b14a9a0bd39843cf8132e067fc48c29e (DEFAULT_TABLE table=rebalance_test_table_16 [id=f40118346c7d461fb1d445e8dab8b379]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.547765 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 96c8dabb170f44fd98bd467f62255713 (DEFAULT_TABLE table=rebalance_test_table_16 [id=f40118346c7d461fb1d445e8dab8b379]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.548030 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96c8dabb170f44fd98bd467f62255713. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.548117 8953 tablet_bootstrap.cc:492] T 96c8dabb170f44fd98bd467f62255713 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:32.548934 8953 tablet_bootstrap.cc:654] T 96c8dabb170f44fd98bd467f62255713 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.549320 8956 tablet_bootstrap.cc:492] T 424123bb8be2460b8855b5a9ad785e57 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:32.550100 7123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b14a9a0bd39843cf8132e067fc48c29e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.550163 8956 tablet_bootstrap.cc:654] T 424123bb8be2460b8855b5a9ad785e57 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.550700 8952 tablet_bootstrap.cc:492] T b14a9a0bd39843cf8132e067fc48c29e P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:32.551409 8956 tablet_bootstrap.cc:492] T 424123bb8be2460b8855b5a9ad785e57 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:32.551424 8953 tablet_bootstrap.cc:492] T 96c8dabb170f44fd98bd467f62255713 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:32.551493 8956 ts_tablet_manager.cc:1367] T 424123bb8be2460b8855b5a9ad785e57 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:32.551517 8953 ts_tablet_manager.cc:1367] T 96c8dabb170f44fd98bd467f62255713 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 01:59:32.551607 8952 tablet_bootstrap.cc:654] T b14a9a0bd39843cf8132e067fc48c29e P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.552168 8953 raft_consensus.cc:348] T 96c8dabb170f44fd98bd467f62255713 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.552165 8956 raft_consensus.cc:348] T 424123bb8be2460b8855b5a9ad785e57 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:32.552325 8956 raft_consensus.cc:374] T 424123bb8be2460b8855b5a9ad785e57 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.552325 8953 raft_consensus.cc:374] T 96c8dabb170f44fd98bd467f62255713 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.552361 8953 raft_consensus.cc:729] T 96c8dabb170f44fd98bd467f62255713 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.552361 8956 raft_consensus.cc:729] T 424123bb8be2460b8855b5a9ad785e57 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.552528 8953 consensus_queue.cc:260] T 96c8dabb170f44fd98bd467f62255713 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.552522 8956 consensus_queue.cc:260] T 424123bb8be2460b8855b5a9ad785e57 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:32.552867 8956 ts_tablet_manager.cc:1398] T 424123bb8be2460b8855b5a9ad785e57 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:32.553110 8956 tablet_bootstrap.cc:492] T b14a9a0bd39843cf8132e067fc48c29e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:32.553134 7255 tablet_service.cc:1467] Processing CreateTablet for tablet 424123bb8be2460b8855b5a9ad785e57 (DEFAULT_TABLE table=rebalance_test_table_16 [id=f40118346c7d461fb1d445e8dab8b379]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.553416 7255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 424123bb8be2460b8855b5a9ad785e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.553702 6993 tablet_service.cc:1467] Processing CreateTablet for tablet 424123bb8be2460b8855b5a9ad785e57 (DEFAULT_TABLE table=rebalance_test_table_16 [id=f40118346c7d461fb1d445e8dab8b379]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:32.553968 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 424123bb8be2460b8855b5a9ad785e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:32.554145 8956 tablet_bootstrap.cc:654] T b14a9a0bd39843cf8132e067fc48c29e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.554857 8953 ts_tablet_manager.cc:1398] T 96c8dabb170f44fd98bd467f62255713 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 01:59:32.555739 8956 tablet_bootstrap.cc:492] T b14a9a0bd39843cf8132e067fc48c29e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:32.555855 8956 ts_tablet_manager.cc:1367] T b14a9a0bd39843cf8132e067fc48c29e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:32.556437 8953 tablet_bootstrap.cc:492] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:32.556383 8956 raft_consensus.cc:348] T b14a9a0bd39843cf8132e067fc48c29e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.556504 8956 raft_consensus.cc:374] T b14a9a0bd39843cf8132e067fc48c29e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.556533 8956 raft_consensus.cc:729] T b14a9a0bd39843cf8132e067fc48c29e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.556666 8956 consensus_queue.cc:260] T b14a9a0bd39843cf8132e067fc48c29e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.556994 8956 ts_tablet_manager.cc:1398] T b14a9a0bd39843cf8132e067fc48c29e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.557976 8953 tablet_bootstrap.cc:654] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.559854 8952 tablet_bootstrap.cc:492] T b14a9a0bd39843cf8132e067fc48c29e P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:32.559981 8952 ts_tablet_manager.cc:1367] T b14a9a0bd39843cf8132e067fc48c29e P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240503 01:59:32.560330 8960 tablet_bootstrap.cc:492] T 96c8dabb170f44fd98bd467f62255713 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:32.560446 8960 ts_tablet_manager.cc:1367] T 96c8dabb170f44fd98bd467f62255713 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 01:59:32.560549 8952 raft_consensus.cc:348] T b14a9a0bd39843cf8132e067fc48c29e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.560680 8952 raft_consensus.cc:374] T b14a9a0bd39843cf8132e067fc48c29e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.560720 8952 raft_consensus.cc:729] T b14a9a0bd39843cf8132e067fc48c29e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.560856 8952 consensus_queue.cc:260] T b14a9a0bd39843cf8132e067fc48c29e P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.560901 8960 raft_consensus.cc:348] T 96c8dabb170f44fd98bd467f62255713 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.561038 8960 raft_consensus.cc:374] T 96c8dabb170f44fd98bd467f62255713 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.561085 8960 raft_consensus.cc:729] T 96c8dabb170f44fd98bd467f62255713 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.561173 8952 ts_tablet_manager.cc:1398] T b14a9a0bd39843cf8132e067fc48c29e P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:32.561218 8960 consensus_queue.cc:260] T 96c8dabb170f44fd98bd467f62255713 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.561369 8952 tablet_bootstrap.cc:492] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:32.561507 8960 ts_tablet_manager.cc:1398] T 96c8dabb170f44fd98bd467f62255713 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:32.562734 8952 tablet_bootstrap.cc:654] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.562731 8960 tablet_bootstrap.cc:492] T 424123bb8be2460b8855b5a9ad785e57 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:32.563802 8952 tablet_bootstrap.cc:492] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:32.563889 8952 ts_tablet_manager.cc:1367] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:59:32.563696 8960 tablet_bootstrap.cc:654] T 424123bb8be2460b8855b5a9ad785e57 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.564391 8952 raft_consensus.cc:348] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.564502 8952 raft_consensus.cc:374] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.564538 8952 raft_consensus.cc:729] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.564653 8952 consensus_queue.cc:260] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.564962 8952 ts_tablet_manager.cc:1398] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:32.566308 8953 tablet_bootstrap.cc:492] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:32.566452 8960 tablet_bootstrap.cc:492] T 424123bb8be2460b8855b5a9ad785e57 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:32.566463 8953 ts_tablet_manager.cc:1367] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20240503 01:59:32.566576 8960 ts_tablet_manager.cc:1367] T 424123bb8be2460b8855b5a9ad785e57 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:32.567021 8953 raft_consensus.cc:348] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.567152 8953 raft_consensus.cc:374] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.567179 8953 raft_consensus.cc:729] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.567255 8960 raft_consensus.cc:348] T 424123bb8be2460b8855b5a9ad785e57 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:32.567296 8953 consensus_queue.cc:260] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.567382 8960 raft_consensus.cc:374] T 424123bb8be2460b8855b5a9ad785e57 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.567432 8960 raft_consensus.cc:729] T 424123bb8be2460b8855b5a9ad785e57 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.567567 8960 consensus_queue.cc:260] T 424123bb8be2460b8855b5a9ad785e57 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:32.567615 8953 ts_tablet_manager.cc:1398] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:32.567831 8960 ts_tablet_manager.cc:1398] T 424123bb8be2460b8855b5a9ad785e57 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:32.568078 8953 tablet_bootstrap.cc:492] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:32.569607 8953 tablet_bootstrap.cc:654] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:32.570844 8953 tablet_bootstrap.cc:492] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:32.570933 8953 ts_tablet_manager.cc:1367] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:32.571437 8953 raft_consensus.cc:348] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:32.571576 8953 raft_consensus.cc:374] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:32.571609 8953 raft_consensus.cc:729] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:32.571748 8953 consensus_queue.cc:260] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:32.572014 8953 ts_tablet_manager.cc:1398] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:32.595502 8940 raft_consensus.cc:482] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:32.595711 8940 raft_consensus.cc:504] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:32.596487 8940 leader_election.cc:288] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:32.596920 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "424123bb8be2460b8855b5a9ad785e57" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:32.597074 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "424123bb8be2460b8855b5a9ad785e57" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:32.597110 7145 raft_consensus.cc:2454] T 424123bb8be2460b8855b5a9ad785e57 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:32.597213 7275 raft_consensus.cc:2454] T 424123bb8be2460b8855b5a9ad785e57 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:32.597473 6949 leader_election.cc:302] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:32.597865 8940 raft_consensus.cc:2781] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:32.597944 8940 raft_consensus.cc:482] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:32.598073 8940 raft_consensus.cc:3037] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.599236 8940 raft_consensus.cc:504] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:32.599577 8940 leader_election.cc:288] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:32.599999 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "424123bb8be2460b8855b5a9ad785e57" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:32.600111 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "424123bb8be2460b8855b5a9ad785e57" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:32.600176 7145 raft_consensus.cc:3037] T 424123bb8be2460b8855b5a9ad785e57 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.600220 7275 raft_consensus.cc:3037] T 424123bb8be2460b8855b5a9ad785e57 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.601266 7145 raft_consensus.cc:2454] T 424123bb8be2460b8855b5a9ad785e57 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:32.601266 7275 raft_consensus.cc:2454] T 424123bb8be2460b8855b5a9ad785e57 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:32.601632 6949 leader_election.cc:302] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:32.601857 8940 raft_consensus.cc:2781] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:32.601954 8940 raft_consensus.cc:686] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:32.602118 8940 consensus_queue.cc:237] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:32.604207 6739 catalog_manager.cc:5484] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:32.619884 8940 raft_consensus.cc:482] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:32.620008 8940 raft_consensus.cc:504] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.620450 8940 leader_election.cc:288] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:32.620725 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14a9a0bd39843cf8132e067fc48c29e" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:32.620857 7145 raft_consensus.cc:2454] T b14a9a0bd39843cf8132e067fc48c29e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:32.620968 6883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14a9a0bd39843cf8132e067fc48c29e" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:32.621131 6883 raft_consensus.cc:2454] T b14a9a0bd39843cf8132e067fc48c29e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:32.621244 6949 leader_election.cc:302] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:32.621420 8940 raft_consensus.cc:2781] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:32.621476 8940 raft_consensus.cc:482] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:32.621529 8940 raft_consensus.cc:3037] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.622383 8940 raft_consensus.cc:504] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.622722 8940 leader_election.cc:288] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:32.623051 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14a9a0bd39843cf8132e067fc48c29e" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:32.623178 7145 raft_consensus.cc:3037] T b14a9a0bd39843cf8132e067fc48c29e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.624132 7145 raft_consensus.cc:2454] T b14a9a0bd39843cf8132e067fc48c29e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:32.624366 6883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14a9a0bd39843cf8132e067fc48c29e" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:32.624433 6949 leader_election.cc:302] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:32.624496 6883 raft_consensus.cc:3037] T b14a9a0bd39843cf8132e067fc48c29e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.624598 8940 raft_consensus.cc:2781] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:32.624778 8940 raft_consensus.cc:686] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:32.624910 8940 consensus_queue.cc:237] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.625451 6883 raft_consensus.cc:2454] T b14a9a0bd39843cf8132e067fc48c29e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:32.627043 6739 catalog_manager.cc:5484] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:32.740339 8397 raft_consensus.cc:482] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:32.740495 8397 raft_consensus.cc:504] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.740999 8397 leader_election.cc:288] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:32.741504 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c8dabb170f44fd98bd467f62255713" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:32.741508 7013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c8dabb170f44fd98bd467f62255713" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:32.741660 7013 raft_consensus.cc:2454] T 96c8dabb170f44fd98bd467f62255713 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:32.741672 7275 raft_consensus.cc:2454] T 96c8dabb170f44fd98bd467f62255713 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:32.742004 6816 leader_election.cc:302] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:32.742187 8397 raft_consensus.cc:2781] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:32.742245 8397 raft_consensus.cc:482] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:32.742300 8397 raft_consensus.cc:3037] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.743340 8397 raft_consensus.cc:504] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.743727 8397 leader_election.cc:288] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:32.744218 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c8dabb170f44fd98bd467f62255713" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:32.744350 7275 raft_consensus.cc:3037] T 96c8dabb170f44fd98bd467f62255713 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.744426 7013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96c8dabb170f44fd98bd467f62255713" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:32.744659 7013 raft_consensus.cc:3037] T 96c8dabb170f44fd98bd467f62255713 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:32.745260 7275 raft_consensus.cc:2454] T 96c8dabb170f44fd98bd467f62255713 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:32.745642 6817 leader_election.cc:302] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:32.745751 7013 raft_consensus.cc:2454] T 96c8dabb170f44fd98bd467f62255713 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:32.745859 8397 raft_consensus.cc:2781] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:32.746021 8397 raft_consensus.cc:686] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 01:59:32.746194 8397 consensus_queue.cc:237] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:32.748498 6739 catalog_manager.cc:5484] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
W20240503 01:59:32.889216 9023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:32.889757 9023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:32.893463 9023 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_16'
I20240503 01:59:33.001252 8873 consensus_queue.cc:1035] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.009595 7145 raft_consensus.cc:1261] T 424123bb8be2460b8855b5a9ad785e57 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.010885 8873 consensus_queue.cc:1035] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.011289 6883 raft_consensus.cc:1261] T b14a9a0bd39843cf8132e067fc48c29e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.011358 7145 raft_consensus.cc:1261] T b14a9a0bd39843cf8132e067fc48c29e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.011926 8940 consensus_queue.cc:1035] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.012426 8940 consensus_queue.cc:1035] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.016208 7276 raft_consensus.cc:1261] T 96c8dabb170f44fd98bd467f62255713 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.016244 7013 raft_consensus.cc:1261] T 96c8dabb170f44fd98bd467f62255713 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.016907 8397 consensus_queue.cc:1035] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.027148 8397 consensus_queue.cc:1035] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 154.51 ms
time per row: 0.077255 ms
I20240503 01:59:33.101924 6737 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:33.108693 7255 tablet_service.cc:1467] Processing CreateTablet for tablet e811662836164c3fb29a1b12ace7eb41 (DEFAULT_TABLE table=rebalance_test_table_17 [id=b061a61957e74150a4af26c16c277330]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.108770 7123 tablet_service.cc:1467] Processing CreateTablet for tablet e811662836164c3fb29a1b12ace7eb41 (DEFAULT_TABLE table=rebalance_test_table_17 [id=b061a61957e74150a4af26c16c277330]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.109050 7255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e811662836164c3fb29a1b12ace7eb41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.109058 7123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e811662836164c3fb29a1b12ace7eb41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.109267 6861 tablet_service.cc:1467] Processing CreateTablet for tablet e811662836164c3fb29a1b12ace7eb41 (DEFAULT_TABLE table=rebalance_test_table_17 [id=b061a61957e74150a4af26c16c277330]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.109542 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e811662836164c3fb29a1b12ace7eb41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.112355 7125 tablet_service.cc:1467] Processing CreateTablet for tablet dec7e3036d384f5db51f50ba93acbbc6 (DEFAULT_TABLE table=rebalance_test_table_17 [id=b061a61957e74150a4af26c16c277330]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.112393 6863 tablet_service.cc:1467] Processing CreateTablet for tablet dec7e3036d384f5db51f50ba93acbbc6 (DEFAULT_TABLE table=rebalance_test_table_17 [id=b061a61957e74150a4af26c16c277330]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.112692 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dec7e3036d384f5db51f50ba93acbbc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.112726 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dec7e3036d384f5db51f50ba93acbbc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.113103 6993 tablet_service.cc:1467] Processing CreateTablet for tablet bf0fe10159d54f0fa922ec74dd96f6a4 (DEFAULT_TABLE table=rebalance_test_table_17 [id=b061a61957e74150a4af26c16c277330]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.113363 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf0fe10159d54f0fa922ec74dd96f6a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.116500 7125 tablet_service.cc:1467] Processing CreateTablet for tablet bf0fe10159d54f0fa922ec74dd96f6a4 (DEFAULT_TABLE table=rebalance_test_table_17 [id=b061a61957e74150a4af26c16c277330]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.116808 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf0fe10159d54f0fa922ec74dd96f6a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.112393 6861 tablet_service.cc:1467] Processing CreateTablet for tablet bf0fe10159d54f0fa922ec74dd96f6a4 (DEFAULT_TABLE table=rebalance_test_table_17 [id=b061a61957e74150a4af26c16c277330]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.118057 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf0fe10159d54f0fa922ec74dd96f6a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.117985 7255 tablet_service.cc:1467] Processing CreateTablet for tablet dec7e3036d384f5db51f50ba93acbbc6 (DEFAULT_TABLE table=rebalance_test_table_17 [id=b061a61957e74150a4af26c16c277330]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.118374 7255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dec7e3036d384f5db51f50ba93acbbc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.120492 9077 tablet_bootstrap.cc:492] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:33.121587 9077 tablet_bootstrap.cc:654] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.122802 9078 tablet_bootstrap.cc:492] T e811662836164c3fb29a1b12ace7eb41 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:33.124332 9077 tablet_bootstrap.cc:492] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:33.124437 9077 ts_tablet_manager.cc:1367] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:33.125190 9077 raft_consensus.cc:348] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:33.125339 9077 raft_consensus.cc:374] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.125370 9077 raft_consensus.cc:729] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.125537 9077 consensus_queue.cc:260] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:33.125870 9077 ts_tablet_manager.cc:1398] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:33.126070 9077 tablet_bootstrap.cc:492] T dec7e3036d384f5db51f50ba93acbbc6 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:33.127270 9080 tablet_bootstrap.cc:492] T bf0fe10159d54f0fa922ec74dd96f6a4 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:33.128170 9077 tablet_bootstrap.cc:654] T dec7e3036d384f5db51f50ba93acbbc6 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.128273 9080 tablet_bootstrap.cc:654] T bf0fe10159d54f0fa922ec74dd96f6a4 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.128608 9078 tablet_bootstrap.cc:654] T e811662836164c3fb29a1b12ace7eb41 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.129701 9077 tablet_bootstrap.cc:492] T dec7e3036d384f5db51f50ba93acbbc6 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:33.129873 9077 ts_tablet_manager.cc:1367] T dec7e3036d384f5db51f50ba93acbbc6 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:33.130376 9077 raft_consensus.cc:348] T dec7e3036d384f5db51f50ba93acbbc6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:33.130555 9077 raft_consensus.cc:374] T dec7e3036d384f5db51f50ba93acbbc6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.130594 9077 raft_consensus.cc:729] T dec7e3036d384f5db51f50ba93acbbc6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.130719 9077 consensus_queue.cc:260] T dec7e3036d384f5db51f50ba93acbbc6 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:33.130770 9079 tablet_bootstrap.cc:492] T e811662836164c3fb29a1b12ace7eb41 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:33.131044 9077 ts_tablet_manager.cc:1398] T dec7e3036d384f5db51f50ba93acbbc6 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:33.131421 9078 tablet_bootstrap.cc:492] T e811662836164c3fb29a1b12ace7eb41 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:33.131590 9078 ts_tablet_manager.cc:1367] T e811662836164c3fb29a1b12ace7eb41 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 01:59:33.131949 9079 tablet_bootstrap.cc:654] T e811662836164c3fb29a1b12ace7eb41 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.132306 9080 tablet_bootstrap.cc:492] T bf0fe10159d54f0fa922ec74dd96f6a4 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:33.132413 9080 ts_tablet_manager.cc:1367] T bf0fe10159d54f0fa922ec74dd96f6a4 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:33.132859 9080 raft_consensus.cc:348] T bf0fe10159d54f0fa922ec74dd96f6a4 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:33.132871 9078 raft_consensus.cc:348] T e811662836164c3fb29a1b12ace7eb41 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:33.132997 9078 raft_consensus.cc:374] T e811662836164c3fb29a1b12ace7eb41 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.133036 9078 raft_consensus.cc:729] T e811662836164c3fb29a1b12ace7eb41 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.133203 9078 consensus_queue.cc:260] T e811662836164c3fb29a1b12ace7eb41 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:33.133514 9078 ts_tablet_manager.cc:1398] T e811662836164c3fb29a1b12ace7eb41 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 01:59:33.133718 9078 tablet_bootstrap.cc:492] T dec7e3036d384f5db51f50ba93acbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:33.133129 9080 raft_consensus.cc:374] T bf0fe10159d54f0fa922ec74dd96f6a4 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.134244 9080 raft_consensus.cc:729] T bf0fe10159d54f0fa922ec74dd96f6a4 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.134721 9080 consensus_queue.cc:260] T bf0fe10159d54f0fa922ec74dd96f6a4 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:33.135143 9080 ts_tablet_manager.cc:1398] T bf0fe10159d54f0fa922ec74dd96f6a4 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 01:59:33.135427 9078 tablet_bootstrap.cc:654] T dec7e3036d384f5db51f50ba93acbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.136682 9078 tablet_bootstrap.cc:492] T dec7e3036d384f5db51f50ba93acbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:33.136684 9079 tablet_bootstrap.cc:492] T e811662836164c3fb29a1b12ace7eb41 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:33.136787 9078 ts_tablet_manager.cc:1367] T dec7e3036d384f5db51f50ba93acbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:33.136791 9079 ts_tablet_manager.cc:1367] T e811662836164c3fb29a1b12ace7eb41 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:33.137284 9079 raft_consensus.cc:348] T e811662836164c3fb29a1b12ace7eb41 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:33.137265 9078 raft_consensus.cc:348] T dec7e3036d384f5db51f50ba93acbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:33.137418 9079 raft_consensus.cc:374] T e811662836164c3fb29a1b12ace7eb41 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.137418 9078 raft_consensus.cc:374] T dec7e3036d384f5db51f50ba93acbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.137463 9079 raft_consensus.cc:729] T e811662836164c3fb29a1b12ace7eb41 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.137463 9078 raft_consensus.cc:729] T dec7e3036d384f5db51f50ba93acbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.137596 9078 consensus_queue.cc:260] T dec7e3036d384f5db51f50ba93acbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:33.137596 9079 consensus_queue.cc:260] T e811662836164c3fb29a1b12ace7eb41 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:33.137877 9078 ts_tablet_manager.cc:1398] T dec7e3036d384f5db51f50ba93acbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:33.137874 9079 ts_tablet_manager.cc:1398] T e811662836164c3fb29a1b12ace7eb41 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:33.138046 9078 tablet_bootstrap.cc:492] T bf0fe10159d54f0fa922ec74dd96f6a4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:33.138046 9079 tablet_bootstrap.cc:492] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:33.139096 9078 tablet_bootstrap.cc:654] T bf0fe10159d54f0fa922ec74dd96f6a4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.139097 9079 tablet_bootstrap.cc:654] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.141331 9078 tablet_bootstrap.cc:492] T bf0fe10159d54f0fa922ec74dd96f6a4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:33.141453 9078 ts_tablet_manager.cc:1367] T bf0fe10159d54f0fa922ec74dd96f6a4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:59:33.141971 9078 raft_consensus.cc:348] T bf0fe10159d54f0fa922ec74dd96f6a4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:33.142124 9078 raft_consensus.cc:374] T bf0fe10159d54f0fa922ec74dd96f6a4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.142202 9078 raft_consensus.cc:729] T bf0fe10159d54f0fa922ec74dd96f6a4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.142351 9078 consensus_queue.cc:260] T bf0fe10159d54f0fa922ec74dd96f6a4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:33.142410 9079 tablet_bootstrap.cc:492] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:33.142511 9079 ts_tablet_manager.cc:1367] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:33.142951 9079 raft_consensus.cc:348] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:33.143060 9079 raft_consensus.cc:374] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.143082 9079 raft_consensus.cc:729] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.143189 9079 consensus_queue.cc:260] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:33.143321 9078 ts_tablet_manager.cc:1398] T bf0fe10159d54f0fa922ec74dd96f6a4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:33.143455 9079 ts_tablet_manager.cc:1398] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:33.143640 9079 tablet_bootstrap.cc:492] T bf0fe10159d54f0fa922ec74dd96f6a4 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:33.144809 9079 tablet_bootstrap.cc:654] T bf0fe10159d54f0fa922ec74dd96f6a4 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.146649 9079 tablet_bootstrap.cc:492] T bf0fe10159d54f0fa922ec74dd96f6a4 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:33.146914 9079 ts_tablet_manager.cc:1367] T bf0fe10159d54f0fa922ec74dd96f6a4 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:33.147498 9079 raft_consensus.cc:348] T bf0fe10159d54f0fa922ec74dd96f6a4 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:33.147863 9079 raft_consensus.cc:374] T bf0fe10159d54f0fa922ec74dd96f6a4 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.147925 9079 raft_consensus.cc:729] T bf0fe10159d54f0fa922ec74dd96f6a4 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.148056 9079 consensus_queue.cc:260] T bf0fe10159d54f0fa922ec74dd96f6a4 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:33.148372 9079 ts_tablet_manager.cc:1398] T bf0fe10159d54f0fa922ec74dd96f6a4 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:33.155259 8933 raft_consensus.cc:482] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:33.155439 8933 raft_consensus.cc:504] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:33.156124 8933 leader_election.cc:288] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:33.156741 9064 raft_consensus.cc:482] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:33.156795 6883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e811662836164c3fb29a1b12ace7eb41" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:33.156916 6883 raft_consensus.cc:2454] T e811662836164c3fb29a1b12ace7eb41 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:33.156879 9064 raft_consensus.cc:504] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:33.157076 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e811662836164c3fb29a1b12ace7eb41" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:33.157225 7145 raft_consensus.cc:2454] T e811662836164c3fb29a1b12ace7eb41 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:33.157235 9064 leader_election.cc:288] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:33.157618 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dec7e3036d384f5db51f50ba93acbbc6" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:33.157721 7145 raft_consensus.cc:2454] T dec7e3036d384f5db51f50ba93acbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:33.157868 7211 leader_election.cc:302] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:33.158139 6818 leader_election.cc:302] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:33.158169 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dec7e3036d384f5db51f50ba93acbbc6" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:33.158258 8933 raft_consensus.cc:2781] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:33.158305 8933 raft_consensus.cc:482] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:33.158308 7275 raft_consensus.cc:2454] T dec7e3036d384f5db51f50ba93acbbc6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:33.158365 8933 raft_consensus.cc:3037] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.158828 9021 raft_consensus.cc:2781] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:33.159026 9021 raft_consensus.cc:482] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:33.159093 9021 raft_consensus.cc:3037] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.159550 8933 raft_consensus.cc:504] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:33.159909 8933 leader_election.cc:288] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Requested vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:33.160435 9021 raft_consensus.cc:504] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:33.160477 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e811662836164c3fb29a1b12ace7eb41" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:33.160593 7145 raft_consensus.cc:3037] T e811662836164c3fb29a1b12ace7eb41 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.160916 9021 leader_election.cc:288] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Requested vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:33.161434 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dec7e3036d384f5db51f50ba93acbbc6" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:33.161433 7144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dec7e3036d384f5db51f50ba93acbbc6" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:33.161562 7144 raft_consensus.cc:3037] T dec7e3036d384f5db51f50ba93acbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.161562 7275 raft_consensus.cc:3037] T dec7e3036d384f5db51f50ba93acbbc6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.161657 6883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e811662836164c3fb29a1b12ace7eb41" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:33.161701 7145 raft_consensus.cc:2454] T e811662836164c3fb29a1b12ace7eb41 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:33.161767 6883 raft_consensus.cc:3037] T e811662836164c3fb29a1b12ace7eb41 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.162463 6883 raft_consensus.cc:2454] T e811662836164c3fb29a1b12ace7eb41 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:33.162477 7144 raft_consensus.cc:2454] T dec7e3036d384f5db51f50ba93acbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:33.162719 7275 raft_consensus.cc:2454] T dec7e3036d384f5db51f50ba93acbbc6 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:33.162876 7211 leader_election.cc:302] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:33.163000 6818 leader_election.cc:302] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:33.163200 9064 raft_consensus.cc:2781] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:33.163273 8933 raft_consensus.cc:2781] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:33.163290 9064 raft_consensus.cc:686] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 01:59:33.163458 8933 raft_consensus.cc:686] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 01:59:33.163470 9064 consensus_queue.cc:237] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:33.163602 8933 consensus_queue.cc:237] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:33.165661 6734 catalog_manager.cc:5484] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:33.166682 6735 catalog_manager.cc:5484] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:33.198447 9004 raft_consensus.cc:482] T bf0fe10159d54f0fa922ec74dd96f6a4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:33.198709 9004 raft_consensus.cc:504] T bf0fe10159d54f0fa922ec74dd96f6a4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:33.199203 9004 leader_election.cc:288] T bf0fe10159d54f0fa922ec74dd96f6a4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:33.199671 7013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0fe10159d54f0fa922ec74dd96f6a4" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:33.199839 7013 raft_consensus.cc:2454] T bf0fe10159d54f0fa922ec74dd96f6a4 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:33.199882 6883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0fe10159d54f0fa922ec74dd96f6a4" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:33.200021 6883 raft_consensus.cc:2454] T bf0fe10159d54f0fa922ec74dd96f6a4 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:33.200359 7081 leader_election.cc:302] T bf0fe10159d54f0fa922ec74dd96f6a4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:33.200603 9092 raft_consensus.cc:2781] T bf0fe10159d54f0fa922ec74dd96f6a4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:33.200726 9092 raft_consensus.cc:482] T bf0fe10159d54f0fa922ec74dd96f6a4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:33.200793 9092 raft_consensus.cc:3037] T bf0fe10159d54f0fa922ec74dd96f6a4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.202312 9092 raft_consensus.cc:504] T bf0fe10159d54f0fa922ec74dd96f6a4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:33.202777 9092 leader_election.cc:288] T bf0fe10159d54f0fa922ec74dd96f6a4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Requested vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:33.203444 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0fe10159d54f0fa922ec74dd96f6a4" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:33.203444 7013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0fe10159d54f0fa922ec74dd96f6a4" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:33.203630 6882 raft_consensus.cc:3037] T bf0fe10159d54f0fa922ec74dd96f6a4 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.203630 7013 raft_consensus.cc:3037] T bf0fe10159d54f0fa922ec74dd96f6a4 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.204736 7013 raft_consensus.cc:2454] T bf0fe10159d54f0fa922ec74dd96f6a4 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:33.204736 6882 raft_consensus.cc:2454] T bf0fe10159d54f0fa922ec74dd96f6a4 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:33.205144 7078 leader_election.cc:302] T bf0fe10159d54f0fa922ec74dd96f6a4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:33.205420 9092 raft_consensus.cc:2781] T bf0fe10159d54f0fa922ec74dd96f6a4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:33.205533 9092 raft_consensus.cc:686] T bf0fe10159d54f0fa922ec74dd96f6a4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Becoming Leader. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Running, Role: LEADER
I20240503 01:59:33.205718 9092 consensus_queue.cc:237] T bf0fe10159d54f0fa922ec74dd96f6a4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:33.207904 6734 catalog_manager.cc:5484] T bf0fe10159d54f0fa922ec74dd96f6a4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
W20240503 01:59:33.365108 9095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:33.365542 9095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:33.370354 9095 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_17'
I20240503 01:59:33.478797 7144 raft_consensus.cc:1261] T e811662836164c3fb29a1b12ace7eb41 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.479632 6883 raft_consensus.cc:1261] T e811662836164c3fb29a1b12ace7eb41 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.479916 7013 raft_consensus.cc:1261] T bf0fe10159d54f0fa922ec74dd96f6a4 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.480126 9003 consensus_queue.cc:1035] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.480355 7275 raft_consensus.cc:1261] T dec7e3036d384f5db51f50ba93acbbc6 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.480813 9003 consensus_queue.cc:1035] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.480955 9064 consensus_queue.cc:1035] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.481606 9093 consensus_queue.cc:1035] T bf0fe10159d54f0fa922ec74dd96f6a4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.481989 7144 raft_consensus.cc:1261] T dec7e3036d384f5db51f50ba93acbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.482388 9064 consensus_queue.cc:1035] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.482745 6883 raft_consensus.cc:1261] T bf0fe10159d54f0fa922ec74dd96f6a4 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:33.484452 9093 consensus_queue.cc:1035] T bf0fe10159d54f0fa922ec74dd96f6a4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:33.514966 9058 mvcc.cc:205] Tried to move back new op lower bound from 7023417644954800128 to 7023417643678724096. Current Snapshot: MvccSnapshot[applied={T|T < 7023417644954800128}]
INSERT report
rows total: 2000
time total: 146.643 ms
time per row: 0.0733214 ms
I20240503 01:59:33.575321 6733 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:33.581892 7255 tablet_service.cc:1467] Processing CreateTablet for tablet 902a8727fa65497aafbc703f3101126b (DEFAULT_TABLE table=rebalance_test_table_18 [id=3a80d416130e4385870aea6e384ed486]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.581928 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 902a8727fa65497aafbc703f3101126b (DEFAULT_TABLE table=rebalance_test_table_18 [id=3a80d416130e4385870aea6e384ed486]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.582253 7255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 902a8727fa65497aafbc703f3101126b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.582265 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 902a8727fa65497aafbc703f3101126b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.584254 9079 tablet_bootstrap.cc:492] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:33.584563 9077 tablet_bootstrap.cc:492] T 902a8727fa65497aafbc703f3101126b P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:33.585216 9079 tablet_bootstrap.cc:654] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.586158 7254 tablet_service.cc:1467] Processing CreateTablet for tablet 249513672b7043d6adcf95ac5ac10a73 (DEFAULT_TABLE table=rebalance_test_table_18 [id=3a80d416130e4385870aea6e384ed486]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.586441 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 249513672b7043d6adcf95ac5ac10a73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.586587 6993 tablet_service.cc:1467] Processing CreateTablet for tablet 902a8727fa65497aafbc703f3101126b (DEFAULT_TABLE table=rebalance_test_table_18 [id=3a80d416130e4385870aea6e384ed486]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.586628 7125 tablet_service.cc:1467] Processing CreateTablet for tablet 556a177350b9483c861972c842a8d193 (DEFAULT_TABLE table=rebalance_test_table_18 [id=3a80d416130e4385870aea6e384ed486]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.586762 6992 tablet_service.cc:1467] Processing CreateTablet for tablet 249513672b7043d6adcf95ac5ac10a73 (DEFAULT_TABLE table=rebalance_test_table_18 [id=3a80d416130e4385870aea6e384ed486]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.586860 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 902a8727fa65497aafbc703f3101126b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.586875 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 556a177350b9483c861972c842a8d193. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.586943 6992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 249513672b7043d6adcf95ac5ac10a73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.586587 6994 tablet_service.cc:1467] Processing CreateTablet for tablet 556a177350b9483c861972c842a8d193 (DEFAULT_TABLE table=rebalance_test_table_18 [id=3a80d416130e4385870aea6e384ed486]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.588179 6994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 556a177350b9483c861972c842a8d193. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.588506 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 249513672b7043d6adcf95ac5ac10a73 (DEFAULT_TABLE table=rebalance_test_table_18 [id=3a80d416130e4385870aea6e384ed486]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.588779 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 249513672b7043d6adcf95ac5ac10a73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.590055 9077 tablet_bootstrap.cc:654] T 902a8727fa65497aafbc703f3101126b P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.591526 9079 tablet_bootstrap.cc:492] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:33.591629 9079 ts_tablet_manager.cc:1367] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:59:33.592021 9078 tablet_bootstrap.cc:492] T 556a177350b9483c861972c842a8d193 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:33.592094 9079 raft_consensus.cc:348] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:33.592211 9079 raft_consensus.cc:374] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.592252 9079 raft_consensus.cc:729] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.592367 9079 consensus_queue.cc:260] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:33.592685 9079 ts_tablet_manager.cc:1398] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:33.592857 9079 tablet_bootstrap.cc:492] T 249513672b7043d6adcf95ac5ac10a73 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:33.592912 9078 tablet_bootstrap.cc:654] T 556a177350b9483c861972c842a8d193 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.593547 9080 tablet_bootstrap.cc:492] T 902a8727fa65497aafbc703f3101126b P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:33.593780 9079 tablet_bootstrap.cc:654] T 249513672b7043d6adcf95ac5ac10a73 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.594496 9080 tablet_bootstrap.cc:654] T 902a8727fa65497aafbc703f3101126b P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.596408 7255 tablet_service.cc:1467] Processing CreateTablet for tablet 556a177350b9483c861972c842a8d193 (DEFAULT_TABLE table=rebalance_test_table_18 [id=3a80d416130e4385870aea6e384ed486]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:33.596694 7255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 556a177350b9483c861972c842a8d193. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:33.596938 9079 tablet_bootstrap.cc:492] T 249513672b7043d6adcf95ac5ac10a73 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:33.597028 9079 ts_tablet_manager.cc:1367] T 249513672b7043d6adcf95ac5ac10a73 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:33.597508 9079 raft_consensus.cc:348] T 249513672b7043d6adcf95ac5ac10a73 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:33.597618 9079 raft_consensus.cc:374] T 249513672b7043d6adcf95ac5ac10a73 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.597651 9079 raft_consensus.cc:729] T 249513672b7043d6adcf95ac5ac10a73 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.597770 9079 consensus_queue.cc:260] T 249513672b7043d6adcf95ac5ac10a73 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:33.598069 9079 ts_tablet_manager.cc:1398] T 249513672b7043d6adcf95ac5ac10a73 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:33.600029 9077 tablet_bootstrap.cc:492] T 902a8727fa65497aafbc703f3101126b P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:33.600162 9077 ts_tablet_manager.cc:1367] T 902a8727fa65497aafbc703f3101126b P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.002s
I20240503 01:59:33.600185 9080 tablet_bootstrap.cc:492] T 902a8727fa65497aafbc703f3101126b P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:33.600317 9080 ts_tablet_manager.cc:1367] T 902a8727fa65497aafbc703f3101126b P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240503 01:59:33.600730 9077 raft_consensus.cc:348] T 902a8727fa65497aafbc703f3101126b P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:33.600850 9077 raft_consensus.cc:374] T 902a8727fa65497aafbc703f3101126b P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.600888 9077 raft_consensus.cc:729] T 902a8727fa65497aafbc703f3101126b P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.600813 9080 raft_consensus.cc:348] T 902a8727fa65497aafbc703f3101126b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:33.600965 9080 raft_consensus.cc:374] T 902a8727fa65497aafbc703f3101126b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.601004 9080 raft_consensus.cc:729] T 902a8727fa65497aafbc703f3101126b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.601014 9077 consensus_queue.cc:260] T 902a8727fa65497aafbc703f3101126b P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:33.601153 9080 consensus_queue.cc:260] T 902a8727fa65497aafbc703f3101126b P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:33.601306 9077 ts_tablet_manager.cc:1398] T 902a8727fa65497aafbc703f3101126b P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:33.601445 9080 ts_tablet_manager.cc:1398] T 902a8727fa65497aafbc703f3101126b P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 01:59:33.601475 9077 tablet_bootstrap.cc:492] T 249513672b7043d6adcf95ac5ac10a73 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:33.601619 9080 tablet_bootstrap.cc:492] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:33.602622 9080 tablet_bootstrap.cc:654] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.602653 9077 tablet_bootstrap.cc:654] T 249513672b7043d6adcf95ac5ac10a73 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.603834 9080 tablet_bootstrap.cc:492] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:33.603905 9080 ts_tablet_manager.cc:1367] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:33.604244 9080 raft_consensus.cc:348] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:33.604328 9080 raft_consensus.cc:374] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.604349 9080 raft_consensus.cc:729] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.604456 9080 consensus_queue.cc:260] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:33.605304 9080 ts_tablet_manager.cc:1398] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:33.605489 9080 tablet_bootstrap.cc:492] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:33.606849 9077 tablet_bootstrap.cc:492] T 249513672b7043d6adcf95ac5ac10a73 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:33.607059 9080 tablet_bootstrap.cc:654] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.607070 9077 ts_tablet_manager.cc:1367] T 249513672b7043d6adcf95ac5ac10a73 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 01:59:33.607678 9077 raft_consensus.cc:348] T 249513672b7043d6adcf95ac5ac10a73 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:33.607892 9077 raft_consensus.cc:374] T 249513672b7043d6adcf95ac5ac10a73 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.607983 9077 raft_consensus.cc:729] T 249513672b7043d6adcf95ac5ac10a73 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.608168 9077 consensus_queue.cc:260] T 249513672b7043d6adcf95ac5ac10a73 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:33.609444 9077 ts_tablet_manager.cc:1398] T 249513672b7043d6adcf95ac5ac10a73 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:33.609776 9077 tablet_bootstrap.cc:492] T 556a177350b9483c861972c842a8d193 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:33.610255 9080 tablet_bootstrap.cc:492] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:33.610471 9080 ts_tablet_manager.cc:1367] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240503 01:59:33.610915 9077 tablet_bootstrap.cc:654] T 556a177350b9483c861972c842a8d193 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:33.611379 9080 raft_consensus.cc:348] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:33.611526 9080 raft_consensus.cc:374] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.611567 9080 raft_consensus.cc:729] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.611688 9080 consensus_queue.cc:260] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:33.612206 9080 ts_tablet_manager.cc:1398] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:33.613042 9078 tablet_bootstrap.cc:492] T 556a177350b9483c861972c842a8d193 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:33.613150 9078 ts_tablet_manager.cc:1367] T 556a177350b9483c861972c842a8d193 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.001s
I20240503 01:59:33.613237 9077 tablet_bootstrap.cc:492] T 556a177350b9483c861972c842a8d193 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:33.613335 9077 ts_tablet_manager.cc:1367] T 556a177350b9483c861972c842a8d193 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:33.613668 9078 raft_consensus.cc:348] T 556a177350b9483c861972c842a8d193 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:33.613763 9078 raft_consensus.cc:374] T 556a177350b9483c861972c842a8d193 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.613785 9078 raft_consensus.cc:729] T 556a177350b9483c861972c842a8d193 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.613757 9077 raft_consensus.cc:348] T 556a177350b9483c861972c842a8d193 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:33.613870 9077 raft_consensus.cc:374] T 556a177350b9483c861972c842a8d193 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:33.613903 9077 raft_consensus.cc:729] T 556a177350b9483c861972c842a8d193 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:33.613901 9078 consensus_queue.cc:260] T 556a177350b9483c861972c842a8d193 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:33.614027 9077 consensus_queue.cc:260] T 556a177350b9483c861972c842a8d193 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:33.614192 9078 ts_tablet_manager.cc:1398] T 556a177350b9483c861972c842a8d193 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:33.614322 9077 ts_tablet_manager.cc:1398] T 556a177350b9483c861972c842a8d193 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:33.671216 9064 raft_consensus.cc:482] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:33.671422 9064 raft_consensus.cc:504] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:33.671994 9064 leader_election.cc:288] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:33.672633 7013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "902a8727fa65497aafbc703f3101126b" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:33.672825 7013 raft_consensus.cc:2454] T 902a8727fa65497aafbc703f3101126b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:33.672991 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "902a8727fa65497aafbc703f3101126b" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:33.673192 7275 raft_consensus.cc:2454] T 902a8727fa65497aafbc703f3101126b P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:33.673189 6816 leader_election.cc:302] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:33.673419 9064 raft_consensus.cc:2781] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:33.673492 9064 raft_consensus.cc:482] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:33.673533 9064 raft_consensus.cc:3037] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.675164 9064 raft_consensus.cc:504] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:33.675631 9064 leader_election.cc:288] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:33.676018 7013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "902a8727fa65497aafbc703f3101126b" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:33.676033 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "902a8727fa65497aafbc703f3101126b" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:33.676177 7013 raft_consensus.cc:3037] T 902a8727fa65497aafbc703f3101126b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.676179 7275 raft_consensus.cc:3037] T 902a8727fa65497aafbc703f3101126b P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.677026 7013 raft_consensus.cc:2454] T 902a8727fa65497aafbc703f3101126b P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:33.677062 7275 raft_consensus.cc:2454] T 902a8727fa65497aafbc703f3101126b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:33.677348 6816 leader_election.cc:302] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:33.677529 9064 raft_consensus.cc:2781] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:33.677624 9064 raft_consensus.cc:686] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 01:59:33.677816 9064 consensus_queue.cc:237] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:33.680104 6734 catalog_manager.cc:5484] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:33.742460 8873 raft_consensus.cc:482] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:33.742628 8873 raft_consensus.cc:504] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:33.743057 8873 leader_election.cc:288] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:33.743430 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "556a177350b9483c861972c842a8d193" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:33.743582 7275 raft_consensus.cc:2454] T 556a177350b9483c861972c842a8d193 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:33.743772 7144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "556a177350b9483c861972c842a8d193" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:33.743961 6948 leader_election.cc:302] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:33.743997 7144 raft_consensus.cc:2454] T 556a177350b9483c861972c842a8d193 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:33.744189 8873 raft_consensus.cc:2781] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:33.744256 8873 raft_consensus.cc:482] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:33.744314 8873 raft_consensus.cc:3037] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.745602 8873 raft_consensus.cc:504] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:33.746012 8873 leader_election.cc:288] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:33.746372 7144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "556a177350b9483c861972c842a8d193" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:33.746382 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "556a177350b9483c861972c842a8d193" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:33.746502 7144 raft_consensus.cc:3037] T 556a177350b9483c861972c842a8d193 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.746549 7275 raft_consensus.cc:3037] T 556a177350b9483c861972c842a8d193 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.747390 7275 raft_consensus.cc:2454] T 556a177350b9483c861972c842a8d193 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:33.747520 7144 raft_consensus.cc:2454] T 556a177350b9483c861972c842a8d193 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:33.747740 6948 leader_election.cc:302] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:33.748039 8873 raft_consensus.cc:2781] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:33.748131 8873 raft_consensus.cc:686] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:33.748287 8873 consensus_queue.cc:237] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:33.750630 6734 catalog_manager.cc:5484] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:33.797889 8873 raft_consensus.cc:482] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:33.798060 8873 raft_consensus.cc:504] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:33.798537 8873 leader_election.cc:288] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:33.799068 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "249513672b7043d6adcf95ac5ac10a73" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:33.799072 6883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "249513672b7043d6adcf95ac5ac10a73" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:33.799255 7275 raft_consensus.cc:2454] T 249513672b7043d6adcf95ac5ac10a73 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:33.799403 6883 raft_consensus.cc:2454] T 249513672b7043d6adcf95ac5ac10a73 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:33.799649 6948 leader_election.cc:302] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:33.799894 8873 raft_consensus.cc:2781] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:33.799968 8873 raft_consensus.cc:482] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:33.800027 8873 raft_consensus.cc:3037] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.801190 8873 raft_consensus.cc:504] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:33.801642 8873 leader_election.cc:288] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:33.801909 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "249513672b7043d6adcf95ac5ac10a73" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:33.802007 6883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "249513672b7043d6adcf95ac5ac10a73" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:33.802111 7275 raft_consensus.cc:3037] T 249513672b7043d6adcf95ac5ac10a73 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.802170 6883 raft_consensus.cc:3037] T 249513672b7043d6adcf95ac5ac10a73 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:33.803031 7275 raft_consensus.cc:2454] T 249513672b7043d6adcf95ac5ac10a73 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:33.803135 6883 raft_consensus.cc:2454] T 249513672b7043d6adcf95ac5ac10a73 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:33.803371 6948 leader_election.cc:302] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:33.803561 8873 raft_consensus.cc:2781] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:33.803653 8873 raft_consensus.cc:686] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:33.803813 8873 consensus_queue.cc:237] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:33.806289 6734 catalog_manager.cc:5484] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
W20240503 01:59:33.982888 9143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:33.983245 9143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:33.986660 9143 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_18'
I20240503 01:59:34.101193 7275 raft_consensus.cc:1261] T 249513672b7043d6adcf95ac5ac10a73 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:34.102470 8873 consensus_queue.cc:1035] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:34.103433 6882 raft_consensus.cc:1261] T 249513672b7043d6adcf95ac5ac10a73 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:34.104956 7144 raft_consensus.cc:1261] T 556a177350b9483c861972c842a8d193 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:34.105069 7274 raft_consensus.cc:1261] T 902a8727fa65497aafbc703f3101126b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:34.105386 9055 consensus_queue.cc:1035] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:34.105427 9021 consensus_queue.cc:1035] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:34.105831 7013 raft_consensus.cc:1261] T 902a8727fa65497aafbc703f3101126b P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:34.109153 9055 consensus_queue.cc:1035] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:34.110242 9064 consensus_queue.cc:1035] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:34.111742 7276 raft_consensus.cc:1261] T 556a177350b9483c861972c842a8d193 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:34.112593 9055 consensus_queue.cc:1035] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:34.141306 9184 mvcc.cc:205] Tried to move back new op lower bound from 7023417647545208832 to 7023417645785624576. Current Snapshot: MvccSnapshot[applied={T|T < 7023417647529246720}]
INSERT report
rows total: 2000
time total: 140.097 ms
time per row: 0.0700486 ms
I20240503 01:59:34.190177 6741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:34.199451 6992 tablet_service.cc:1467] Processing CreateTablet for tablet c4ac4ad7f9eb4723814f65ce96758c0e (DEFAULT_TABLE table=rebalance_test_table_19 [id=d2d157b43e4243979ed58dfe4b2fe341]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:34.199534 6863 tablet_service.cc:1467] Processing CreateTablet for tablet dfdcdd54606e4320a5d0f506eb6dabf8 (DEFAULT_TABLE table=rebalance_test_table_19 [id=d2d157b43e4243979ed58dfe4b2fe341]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:34.199815 6992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ac4ad7f9eb4723814f65ce96758c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:34.199815 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfdcdd54606e4320a5d0f506eb6dabf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:34.199957 6861 tablet_service.cc:1467] Processing CreateTablet for tablet c4ac4ad7f9eb4723814f65ce96758c0e (DEFAULT_TABLE table=rebalance_test_table_19 [id=d2d157b43e4243979ed58dfe4b2fe341]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:34.200333 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ac4ad7f9eb4723814f65ce96758c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:34.200482 7123 tablet_service.cc:1467] Processing CreateTablet for tablet 4be54233e31248cd8deafa6b7845a3f8 (DEFAULT_TABLE table=rebalance_test_table_19 [id=d2d157b43e4243979ed58dfe4b2fe341]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:34.200778 7123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4be54233e31248cd8deafa6b7845a3f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:34.202407 6994 tablet_service.cc:1467] Processing CreateTablet for tablet 4be54233e31248cd8deafa6b7845a3f8 (DEFAULT_TABLE table=rebalance_test_table_19 [id=d2d157b43e4243979ed58dfe4b2fe341]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:34.202716 6994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4be54233e31248cd8deafa6b7845a3f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:34.202901 7124 tablet_service.cc:1467] Processing CreateTablet for tablet dfdcdd54606e4320a5d0f506eb6dabf8 (DEFAULT_TABLE table=rebalance_test_table_19 [id=d2d157b43e4243979ed58dfe4b2fe341]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:34.203203 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfdcdd54606e4320a5d0f506eb6dabf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:34.203891 9204 tablet_bootstrap.cc:492] T 4be54233e31248cd8deafa6b7845a3f8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:34.204797 9204 tablet_bootstrap.cc:654] T 4be54233e31248cd8deafa6b7845a3f8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:34.205065 7255 tablet_service.cc:1467] Processing CreateTablet for tablet 4be54233e31248cd8deafa6b7845a3f8 (DEFAULT_TABLE table=rebalance_test_table_19 [id=d2d157b43e4243979ed58dfe4b2fe341]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:34.205363 7255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4be54233e31248cd8deafa6b7845a3f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:34.205605 7254 tablet_service.cc:1467] Processing CreateTablet for tablet dfdcdd54606e4320a5d0f506eb6dabf8 (DEFAULT_TABLE table=rebalance_test_table_19 [id=d2d157b43e4243979ed58dfe4b2fe341]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:34.205721 9204 tablet_bootstrap.cc:492] T 4be54233e31248cd8deafa6b7845a3f8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:34.205785 9204 ts_tablet_manager.cc:1367] T 4be54233e31248cd8deafa6b7845a3f8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:34.205816 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfdcdd54606e4320a5d0f506eb6dabf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:34.206130 9204 raft_consensus.cc:348] T 4be54233e31248cd8deafa6b7845a3f8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:34.206218 9204 raft_consensus.cc:374] T 4be54233e31248cd8deafa6b7845a3f8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:34.206238 9204 raft_consensus.cc:729] T 4be54233e31248cd8deafa6b7845a3f8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:34.206365 9204 consensus_queue.cc:260] T 4be54233e31248cd8deafa6b7845a3f8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:34.206735 9204 ts_tablet_manager.cc:1398] T 4be54233e31248cd8deafa6b7845a3f8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:34.207580 9203 tablet_bootstrap.cc:492] T c4ac4ad7f9eb4723814f65ce96758c0e P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:34.208197 7125 tablet_service.cc:1467] Processing CreateTablet for tablet c4ac4ad7f9eb4723814f65ce96758c0e (DEFAULT_TABLE table=rebalance_test_table_19 [id=d2d157b43e4243979ed58dfe4b2fe341]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:34.208482 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ac4ad7f9eb4723814f65ce96758c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:34.209041 9203 tablet_bootstrap.cc:654] T c4ac4ad7f9eb4723814f65ce96758c0e P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:34.209785 9202 tablet_bootstrap.cc:492] T dfdcdd54606e4320a5d0f506eb6dabf8 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:34.211081 9202 tablet_bootstrap.cc:654] T dfdcdd54606e4320a5d0f506eb6dabf8 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:34.211172 9203 tablet_bootstrap.cc:492] T c4ac4ad7f9eb4723814f65ce96758c0e P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:34.211295 9203 ts_tablet_manager.cc:1367] T c4ac4ad7f9eb4723814f65ce96758c0e P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:34.211828 9203 raft_consensus.cc:348] T c4ac4ad7f9eb4723814f65ce96758c0e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:34.211966 9203 raft_consensus.cc:374] T c4ac4ad7f9eb4723814f65ce96758c0e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:34.212013 9203 raft_consensus.cc:729] T c4ac4ad7f9eb4723814f65ce96758c0e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:34.212146 9203 consensus_queue.cc:260] T c4ac4ad7f9eb4723814f65ce96758c0e P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:34.212436 9204 tablet_bootstrap.cc:492] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:34.212476 9203 ts_tablet_manager.cc:1398] T c4ac4ad7f9eb4723814f65ce96758c0e P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:34.212659 9203 tablet_bootstrap.cc:492] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:34.213594 9204 tablet_bootstrap.cc:654] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:34.213721 9203 tablet_bootstrap.cc:654] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:34.215209 9204 tablet_bootstrap.cc:492] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:34.215359 9204 ts_tablet_manager.cc:1367] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:34.215358 9206 tablet_bootstrap.cc:492] T 4be54233e31248cd8deafa6b7845a3f8 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:34.215919 9204 raft_consensus.cc:348] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:34.216101 9204 raft_consensus.cc:374] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:34.216435 9204 raft_consensus.cc:729] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:34.216524 9206 tablet_bootstrap.cc:654] T 4be54233e31248cd8deafa6b7845a3f8 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:34.216615 9204 consensus_queue.cc:260] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:34.217144 9204 ts_tablet_manager.cc:1398] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:34.217420 9204 tablet_bootstrap.cc:492] T c4ac4ad7f9eb4723814f65ce96758c0e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:34.217725 9206 tablet_bootstrap.cc:492] T 4be54233e31248cd8deafa6b7845a3f8 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:34.217849 9206 ts_tablet_manager.cc:1367] T 4be54233e31248cd8deafa6b7845a3f8 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:34.217868 9202 tablet_bootstrap.cc:492] T dfdcdd54606e4320a5d0f506eb6dabf8 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:34.217998 9202 ts_tablet_manager.cc:1367] T dfdcdd54606e4320a5d0f506eb6dabf8 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 01:59:34.218433 9204 tablet_bootstrap.cc:654] T c4ac4ad7f9eb4723814f65ce96758c0e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:34.218611 9202 raft_consensus.cc:348] T dfdcdd54606e4320a5d0f506eb6dabf8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:34.218669 9206 raft_consensus.cc:348] T 4be54233e31248cd8deafa6b7845a3f8 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:34.218832 9206 raft_consensus.cc:374] T 4be54233e31248cd8deafa6b7845a3f8 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:34.218885 9206 raft_consensus.cc:729] T 4be54233e31248cd8deafa6b7845a3f8 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:34.219036 9206 consensus_queue.cc:260] T 4be54233e31248cd8deafa6b7845a3f8 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:34.219069 9202 raft_consensus.cc:374] T dfdcdd54606e4320a5d0f506eb6dabf8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:34.219310 9202 raft_consensus.cc:729] T dfdcdd54606e4320a5d0f506eb6dabf8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:34.219368 9206 ts_tablet_manager.cc:1398] T 4be54233e31248cd8deafa6b7845a3f8 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:34.219590 9206 tablet_bootstrap.cc:492] T dfdcdd54606e4320a5d0f506eb6dabf8 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:34.219594 9202 consensus_queue.cc:260] T dfdcdd54606e4320a5d0f506eb6dabf8 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:34.220029 9202 ts_tablet_manager.cc:1398] T dfdcdd54606e4320a5d0f506eb6dabf8 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:34.220528 9206 tablet_bootstrap.cc:654] T dfdcdd54606e4320a5d0f506eb6dabf8 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:34.220923 9202 tablet_bootstrap.cc:492] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:34.222862 9206 tablet_bootstrap.cc:492] T dfdcdd54606e4320a5d0f506eb6dabf8 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:34.222870 9202 tablet_bootstrap.cc:654] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:34.223037 9206 ts_tablet_manager.cc:1367] T dfdcdd54606e4320a5d0f506eb6dabf8 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:34.223555 9206 raft_consensus.cc:348] T dfdcdd54606e4320a5d0f506eb6dabf8 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:34.223704 9206 raft_consensus.cc:374] T dfdcdd54606e4320a5d0f506eb6dabf8 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:34.223769 9206 raft_consensus.cc:729] T dfdcdd54606e4320a5d0f506eb6dabf8 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:34.223915 9206 consensus_queue.cc:260] T dfdcdd54606e4320a5d0f506eb6dabf8 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:34.224386 9206 ts_tablet_manager.cc:1398] T dfdcdd54606e4320a5d0f506eb6dabf8 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:34.224656 9203 tablet_bootstrap.cc:492] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:34.224817 9203 ts_tablet_manager.cc:1367] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 01:59:34.225390 9203 raft_consensus.cc:348] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:34.225520 9203 raft_consensus.cc:374] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:34.225554 9203 raft_consensus.cc:729] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:34.225683 9203 consensus_queue.cc:260] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:34.226033 9203 ts_tablet_manager.cc:1398] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:34.226795 9204 tablet_bootstrap.cc:492] T c4ac4ad7f9eb4723814f65ce96758c0e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:34.226893 9204 ts_tablet_manager.cc:1367] T c4ac4ad7f9eb4723814f65ce96758c0e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 01:59:34.227149 9202 tablet_bootstrap.cc:492] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:34.227244 9202 ts_tablet_manager.cc:1367] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:34.227365 9204 raft_consensus.cc:348] T c4ac4ad7f9eb4723814f65ce96758c0e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:34.227473 9204 raft_consensus.cc:374] T c4ac4ad7f9eb4723814f65ce96758c0e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:34.227511 9204 raft_consensus.cc:729] T c4ac4ad7f9eb4723814f65ce96758c0e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:34.227632 9204 consensus_queue.cc:260] T c4ac4ad7f9eb4723814f65ce96758c0e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:34.227757 9202 raft_consensus.cc:348] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:34.227849 9202 raft_consensus.cc:374] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:34.227869 9202 raft_consensus.cc:729] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:34.227901 9204 ts_tablet_manager.cc:1398] T c4ac4ad7f9eb4723814f65ce96758c0e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:34.227988 9202 consensus_queue.cc:260] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:34.229113 9202 ts_tablet_manager.cc:1398] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:34.250432 9055 raft_consensus.cc:482] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:34.250684 9055 raft_consensus.cc:504] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:34.251243 9055 leader_election.cc:288] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:34.251922 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be54233e31248cd8deafa6b7845a3f8" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:34.252007 7144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be54233e31248cd8deafa6b7845a3f8" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:34.252131 7274 raft_consensus.cc:2454] T 4be54233e31248cd8deafa6b7845a3f8 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:34.252194 7144 raft_consensus.cc:2454] T 4be54233e31248cd8deafa6b7845a3f8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:34.252566 6949 leader_election.cc:302] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:34.252765 9055 raft_consensus.cc:2781] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:34.252830 9055 raft_consensus.cc:482] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:34.252868 9055 raft_consensus.cc:3037] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.253928 9055 raft_consensus.cc:504] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:34.254334 9055 leader_election.cc:288] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:34.254702 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be54233e31248cd8deafa6b7845a3f8" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:34.254825 7145 raft_consensus.cc:3037] T 4be54233e31248cd8deafa6b7845a3f8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.254786 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be54233e31248cd8deafa6b7845a3f8" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:34.254948 7274 raft_consensus.cc:3037] T 4be54233e31248cd8deafa6b7845a3f8 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.255681 7145 raft_consensus.cc:2454] T 4be54233e31248cd8deafa6b7845a3f8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:34.255853 7274 raft_consensus.cc:2454] T 4be54233e31248cd8deafa6b7845a3f8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:34.256037 6949 leader_election.cc:302] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:34.256266 9055 raft_consensus.cc:2781] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:34.256346 9055 raft_consensus.cc:686] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:34.256489 9055 consensus_queue.cc:237] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:34.259111 6741 catalog_manager.cc:5484] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:34.329702 9092 raft_consensus.cc:482] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:34.329849 9092 raft_consensus.cc:504] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:34.330307 9092 leader_election.cc:288] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:34.330794 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfdcdd54606e4320a5d0f506eb6dabf8" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:34.330825 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfdcdd54606e4320a5d0f506eb6dabf8" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:34.330967 7274 raft_consensus.cc:2454] T dfdcdd54606e4320a5d0f506eb6dabf8 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:34.330974 6882 raft_consensus.cc:2454] T dfdcdd54606e4320a5d0f506eb6dabf8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:34.331363 7079 leader_election.cc:302] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:34.331622 9092 raft_consensus.cc:2781] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:34.331673 9092 raft_consensus.cc:482] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:34.331717 9092 raft_consensus.cc:3037] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.332615 9092 raft_consensus.cc:504] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:34.332998 9092 leader_election.cc:288] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:34.333554 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfdcdd54606e4320a5d0f506eb6dabf8" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:34.333678 6882 raft_consensus.cc:3037] T dfdcdd54606e4320a5d0f506eb6dabf8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.333884 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfdcdd54606e4320a5d0f506eb6dabf8" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:34.334008 7274 raft_consensus.cc:3037] T dfdcdd54606e4320a5d0f506eb6dabf8 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.334888 6882 raft_consensus.cc:2454] T dfdcdd54606e4320a5d0f506eb6dabf8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:34.335008 7274 raft_consensus.cc:2454] T dfdcdd54606e4320a5d0f506eb6dabf8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:34.335326 7079 leader_election.cc:302] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:34.335507 9092 raft_consensus.cc:2781] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:34.335749 9092 raft_consensus.cc:686] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Becoming Leader. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Running, Role: LEADER
I20240503 01:59:34.335913 9092 consensus_queue.cc:237] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:34.338474 6741 catalog_manager.cc:5484] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:34.512163 9021 raft_consensus.cc:482] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:34.512310 9021 raft_consensus.cc:504] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:34.512799 9021 leader_election.cc:288] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:34.513082 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ac4ad7f9eb4723814f65ce96758c0e" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:34.513244 7145 raft_consensus.cc:2454] T c4ac4ad7f9eb4723814f65ce96758c0e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:34.513242 7013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ac4ad7f9eb4723814f65ce96758c0e" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:34.513533 7013 raft_consensus.cc:2454] T c4ac4ad7f9eb4723814f65ce96758c0e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:34.513721 6818 leader_election.cc:302] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:34.513933 9021 raft_consensus.cc:2781] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:34.513983 9021 raft_consensus.cc:482] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:34.514024 9021 raft_consensus.cc:3037] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.515188 9021 raft_consensus.cc:504] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:34.515587 9021 leader_election.cc:288] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Requested vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:34.516013 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ac4ad7f9eb4723814f65ce96758c0e" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:34.516099 7013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ac4ad7f9eb4723814f65ce96758c0e" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:34.516191 7013 raft_consensus.cc:3037] T c4ac4ad7f9eb4723814f65ce96758c0e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.516182 7145 raft_consensus.cc:3037] T c4ac4ad7f9eb4723814f65ce96758c0e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:34.517323 7145 raft_consensus.cc:2454] T c4ac4ad7f9eb4723814f65ce96758c0e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:34.517344 7013 raft_consensus.cc:2454] T c4ac4ad7f9eb4723814f65ce96758c0e P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:34.517660 6818 leader_election.cc:302] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:34.517971 9021 raft_consensus.cc:2781] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:34.518114 9021 raft_consensus.cc:686] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 01:59:34.518247 9021 consensus_queue.cc:237] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:34.520715 6741 catalog_manager.cc:5484] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:34.678953 9055 consensus_queue.cc:1035] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:34.682511 9055 consensus_queue.cc:1035] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 01:59:34.708334 9219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:34.708642 9219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:34.712100 9219 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_19'
I20240503 01:59:34.814402 6882 raft_consensus.cc:1261] T dfdcdd54606e4320a5d0f506eb6dabf8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:34.815198 9092 consensus_queue.cc:1035] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:34.817140 7275 raft_consensus.cc:1261] T dfdcdd54606e4320a5d0f506eb6dabf8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:34.817798 9216 consensus_queue.cc:1035] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:34.825227 7014 raft_consensus.cc:1261] T c4ac4ad7f9eb4723814f65ce96758c0e P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:34.825222 7145 raft_consensus.cc:1261] T c4ac4ad7f9eb4723814f65ce96758c0e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:34.825899 9021 consensus_queue.cc:1035] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:34.827442 9021 consensus_queue.cc:1035] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 126.837 ms
time per row: 0.0634183 ms
I20240503 01:59:34.885736 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:37387
--local_ip_for_outbound_sockets=127.0.99.133
--tserver_master_addrs=127.0.99.190:36303
--webserver_port=37949
--webserver_interface=127.0.99.133
--builtin_ntp_servers=127.0.99.148:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 01:59:35.017449 9281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:35.017756 9281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:35.017814 9281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:35.021378 9281 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:59:35.021469 9281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:35.021557 9281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 01:59:35.025815 9281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:37387
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=37949
--tserver_master_addrs=127.0.99.190:36303
--never_fsync=true
--heap_profile_path=/tmp/kudu.9281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:35.027022 9281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:35.028262 9281 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:35.038079 9286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:35.039168 9287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:35.040383 9289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:35.041539 9281 server_base.cc:1034] running on GCE node
I20240503 01:59:35.042172 9281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:35.043040 9281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:35.044220 9281 hybrid_clock.cc:648] HybridClock initialized: now 1714701575044182 us; error 57 us; skew 500 ppm
I20240503 01:59:35.046728 9281 webserver.cc:460] Webserver started at http://127.0.99.133:37949/ using document root <none> and password file <none>
I20240503 01:59:35.047360 9281 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:35.047435 9281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:35.050992 9281 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20240503 01:59:35.053412 9295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:35.054962 9281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:35.055087 9281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "12e8c683db054df3b75c6285b98a5ad8"
format_stamp: "Formatted at 2024-05-03 01:59:18 on dist-test-slave-w9tw"
I20240503 01:59:35.055610 9281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:35.069300 9281 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:35.069972 9281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:35.070123 9281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:35.070734 9281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:35.071815 9281 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:35.071869 9281 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:35.071913 9281 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:35.071952 9281 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:35.081913 9281 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:37387
I20240503 01:59:35.082124 9408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:37387 every 8 connection(s)
I20240503 01:59:35.083130 9281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 01:59:35.086316 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 9281
I20240503 01:59:35.086802 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:43471
--local_ip_for_outbound_sockets=127.0.99.134
--tserver_master_addrs=127.0.99.190:36303
--webserver_port=38403
--webserver_interface=127.0.99.134
--builtin_ntp_servers=127.0.99.148:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 01:59:35.093495 9409 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36303
I20240503 01:59:35.093767 9409 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:35.094336 9409 heartbeater.cc:511] Master 127.0.99.190:36303 requested a full tablet report, sending...
I20240503 01:59:35.095626 6741 ts_manager.cc:194] Re-registered known tserver with Master: 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:35.096380 6741 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:47239
W20240503 01:59:35.229074 9412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:35.229343 9412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:35.229418 9412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:35.232925 9412 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:59:35.233028 9412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:35.233143 9412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 01:59:35.237372 9412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:43471
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=38403
--tserver_master_addrs=127.0.99.190:36303
--never_fsync=true
--heap_profile_path=/tmp/kudu.9412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:35.238428 9412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:35.239729 9412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:35.250303 9417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:35.251142 9418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:35.252424 9412 server_base.cc:1034] running on GCE node
W20240503 01:59:35.253381 9420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:35.253938 9412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:35.254704 9412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:35.255883 9412 hybrid_clock.cc:648] HybridClock initialized: now 1714701575255868 us; error 29 us; skew 500 ppm
I20240503 01:59:35.258100 9412 webserver.cc:460] Webserver started at http://127.0.99.134:38403/ using document root <none> and password file <none>
I20240503 01:59:35.258733 9412 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:35.258819 9412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:35.262570 9412 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:35.264806 9426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:35.266079 9412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:35.266207 9412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "bb5364d730cb4489a0bfc6c270b7450a"
format_stamp: "Formatted at 2024-05-03 01:59:18 on dist-test-slave-w9tw"
I20240503 01:59:35.266796 9412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:35.312355 9412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:35.313036 9412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:35.313207 9412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:35.313886 9412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:35.314975 9412 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:35.315042 9412 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:35.315097 9412 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:35.315131 9412 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:35.325033 9412 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:43471
I20240503 01:59:35.325048 9539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:43471 every 8 connection(s)
I20240503 01:59:35.326375 9412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 01:59:35.330971 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 9412
I20240503 01:59:35.331362 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:46087
--local_ip_for_outbound_sockets=127.0.99.135
--tserver_master_addrs=127.0.99.190:36303
--webserver_port=41755
--webserver_interface=127.0.99.135
--builtin_ntp_servers=127.0.99.148:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 01:59:35.336863 9540 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36303
I20240503 01:59:35.337146 9540 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:35.337775 9540 heartbeater.cc:511] Master 127.0.99.190:36303 requested a full tablet report, sending...
I20240503 01:59:35.338991 6741 ts_manager.cc:194] Re-registered known tserver with Master: bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:35.339548 6741 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:39315
W20240503 01:59:35.460757 9543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:35.461076 9543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:35.461140 9543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:35.465015 9543 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:59:35.465124 9543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:35.465281 9543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 01:59:35.469942 9543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:46087
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=41755
--tserver_master_addrs=127.0.99.190:36303
--never_fsync=true
--heap_profile_path=/tmp/kudu.9543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:35.471325 9543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:35.472925 9543 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:35.484063 9550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:35.484712 9549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:35.485867 9552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:35.486970 9543 server_base.cc:1034] running on GCE node
I20240503 01:59:35.487514 9543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:35.488263 9543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:35.489480 9543 hybrid_clock.cc:648] HybridClock initialized: now 1714701575489445 us; error 41 us; skew 500 ppm
I20240503 01:59:35.492234 9543 webserver.cc:460] Webserver started at http://127.0.99.135:41755/ using document root <none> and password file <none>
I20240503 01:59:35.492944 9543 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:35.493029 9543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:35.497246 9543 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20240503 01:59:35.499567 9558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:35.500725 9543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 01:59:35.500833 9543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
format_stamp: "Formatted at 2024-05-03 01:59:19 on dist-test-slave-w9tw"
I20240503 01:59:35.501358 9543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:35.520764 9543 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:35.521407 9543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:35.521561 9543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:35.522074 9543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:35.523162 9543 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:35.523218 9543 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:35.523278 9543 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:35.523309 9543 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:35.532913 9543 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.135:46087
I20240503 01:59:35.532938 9671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.135:46087 every 8 connection(s)
I20240503 01:59:35.534020 9543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 01:59:35.538000 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 9543
I20240503 01:59:35.538509 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.136:42053
--local_ip_for_outbound_sockets=127.0.99.136
--tserver_master_addrs=127.0.99.190:36303
--webserver_port=46033
--webserver_interface=127.0.99.136
--builtin_ntp_servers=127.0.99.148:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 01:59:35.544656 9672 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36303
I20240503 01:59:35.544914 9672 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:35.545424 9672 heartbeater.cc:511] Master 127.0.99.190:36303 requested a full tablet report, sending...
I20240503 01:59:35.546496 6741 ts_manager.cc:194] Re-registered known tserver with Master: b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:35.547156 6741 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.135:56545
W20240503 01:59:35.662597 9675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:35.662846 9675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 01:59:35.662894 9675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:35.666190 9675 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 01:59:35.666275 9675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 01:59:35.666348 9675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.136
I20240503 01:59:35.670397 9675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.136:42053
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.99.136
--webserver_port=46033
--tserver_master_addrs=127.0.99.190:36303
--never_fsync=true
--heap_profile_path=/tmp/kudu.9675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.136
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 01:59:35.671442 9675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 01:59:35.673000 9675 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 01:59:35.682804 9681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:35.683430 9680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 01:59:35.684712 9683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 01:59:35.685751 9675 server_base.cc:1034] running on GCE node
I20240503 01:59:35.686197 9675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 01:59:35.686863 9675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 01:59:35.688057 9675 hybrid_clock.cc:648] HybridClock initialized: now 1714701575688039 us; error 37 us; skew 500 ppm
I20240503 01:59:35.690353 9675 webserver.cc:460] Webserver started at http://127.0.99.136:46033/ using document root <none> and password file <none>
I20240503 01:59:35.690999 9675 fs_manager.cc:359] Metadata directory not provided
I20240503 01:59:35.691072 9675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 01:59:35.696091 9675 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20240503 01:59:35.699312 9689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:35.700752 9675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:35.700874 9675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "db1a0000aaa346f79c0aa4b6c5105191"
format_stamp: "Formatted at 2024-05-03 01:59:19 on dist-test-slave-w9tw"
I20240503 01:59:35.701351 9675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 01:59:35.737880 9675 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 01:59:35.738554 9675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 01:59:35.738694 9675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 01:59:35.739205 9675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 01:59:35.740164 9675 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 01:59:35.740216 9675 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:35.740257 9675 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 01:59:35.740270 9675 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 01:59:35.749261 9675 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.136:42053
I20240503 01:59:35.749408 9802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.136:42053 every 8 connection(s)
I20240503 01:59:35.750571 9675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 01:59:35.756716 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 9675
I20240503 01:59:35.760353 6741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:35.762893 9803 heartbeater.cc:347] Connected to a master server at 127.0.99.190:36303
I20240503 01:59:35.763175 9803 heartbeater.cc:464] Registering TS with master...
I20240503 01:59:35.763835 9803 heartbeater.cc:511] Master 127.0.99.190:36303 requested a full tablet report, sending...
I20240503 01:59:35.764839 6735 ts_manager.cc:194] Re-registered known tserver with Master: db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:35.765563 6735 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.136:34799
I20240503 01:59:35.776060 6994 tablet_service.cc:1467] Processing CreateTablet for tablet 3fea2c89a1f2426c83ea043339dbe3c9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b8e14b57de9c493d97360942327b846d]), partition=
I20240503 01:59:35.776240 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 3fea2c89a1f2426c83ea043339dbe3c9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b8e14b57de9c493d97360942327b846d]), partition=
I20240503 01:59:35.776356 6994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fea2c89a1f2426c83ea043339dbe3c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:35.776525 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fea2c89a1f2426c83ea043339dbe3c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:35.778856 9818 tablet_bootstrap.cc:492] T 3fea2c89a1f2426c83ea043339dbe3c9 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:35.779834 9818 tablet_bootstrap.cc:654] T 3fea2c89a1f2426c83ea043339dbe3c9 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:35.782289 9818 tablet_bootstrap.cc:492] T 3fea2c89a1f2426c83ea043339dbe3c9 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:35.782469 9818 ts_tablet_manager.cc:1367] T 3fea2c89a1f2426c83ea043339dbe3c9 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:35.783098 9818 raft_consensus.cc:348] T 3fea2c89a1f2426c83ea043339dbe3c9 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:35.783273 9818 raft_consensus.cc:374] T 3fea2c89a1f2426c83ea043339dbe3c9 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:35.783305 9818 raft_consensus.cc:729] T 3fea2c89a1f2426c83ea043339dbe3c9 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:35.783504 9818 consensus_queue.cc:260] T 3fea2c89a1f2426c83ea043339dbe3c9 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:35.783902 9818 ts_tablet_manager.cc:1398] T 3fea2c89a1f2426c83ea043339dbe3c9 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:35.784819 9343 tablet_service.cc:1467] Processing CreateTablet for tablet 3fea2c89a1f2426c83ea043339dbe3c9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b8e14b57de9c493d97360942327b846d]), partition=
I20240503 01:59:35.785192 9817 tablet_bootstrap.cc:492] T 3fea2c89a1f2426c83ea043339dbe3c9 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:35.785625 9343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fea2c89a1f2426c83ea043339dbe3c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:35.786356 9817 tablet_bootstrap.cc:654] T 3fea2c89a1f2426c83ea043339dbe3c9 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:35.787917 9817 tablet_bootstrap.cc:492] T 3fea2c89a1f2426c83ea043339dbe3c9 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:35.788048 9817 ts_tablet_manager.cc:1367] T 3fea2c89a1f2426c83ea043339dbe3c9 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:35.788560 9817 raft_consensus.cc:348] T 3fea2c89a1f2426c83ea043339dbe3c9 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:35.788673 9817 raft_consensus.cc:374] T 3fea2c89a1f2426c83ea043339dbe3c9 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:35.788700 9817 raft_consensus.cc:729] T 3fea2c89a1f2426c83ea043339dbe3c9 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:35.788832 9817 consensus_queue.cc:260] T 3fea2c89a1f2426c83ea043339dbe3c9 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:35.789144 9817 ts_tablet_manager.cc:1398] T 3fea2c89a1f2426c83ea043339dbe3c9 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:35.794404 9821 tablet_bootstrap.cc:492] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:35.797639 9821 tablet_bootstrap.cc:654] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:35.798444 9821 log.cc:824] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:35.800096 9821 tablet_bootstrap.cc:492] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:35.800272 9821 ts_tablet_manager.cc:1367] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 01:59:35.803515 9821 raft_consensus.cc:348] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:35.804044 9821 raft_consensus.cc:374] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:35.804103 9821 raft_consensus.cc:729] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:35.804565 9821 consensus_queue.cc:260] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:35.805348 9409 heartbeater.cc:503] Master 127.0.99.190:36303 was elected leader, sending a full tablet report...
I20240503 01:59:35.805773 9821 ts_tablet_manager.cc:1398] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20240503 01:59:35.834546 9410 tablet.cc:2369] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:36.003599 9823 raft_consensus.cc:482] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:36.003800 9823 raft_consensus.cc:504] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:36.005081 9823 leader_election.cc:288] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:36.011659 6883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fea2c89a1f2426c83ea043339dbe3c9" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:36.011855 6883 raft_consensus.cc:2454] T 3fea2c89a1f2426c83ea043339dbe3c9 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:36.012635 9299 leader_election.cc:302] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:36.013108 9823 raft_consensus.cc:2781] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:36.013188 9823 raft_consensus.cc:482] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:36.013228 9823 raft_consensus.cc:3037] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:36.014741 9823 raft_consensus.cc:504] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:36.015164 9823 leader_election.cc:288] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Requested vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:36.016649 7013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fea2c89a1f2426c83ea043339dbe3c9" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:36.016810 7013 raft_consensus.cc:2454] T 3fea2c89a1f2426c83ea043339dbe3c9 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:36.016963 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fea2c89a1f2426c83ea043339dbe3c9" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:36.017103 7014 raft_consensus.cc:3037] T 3fea2c89a1f2426c83ea043339dbe3c9 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:36.018098 6883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fea2c89a1f2426c83ea043339dbe3c9" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:36.018246 6883 raft_consensus.cc:3037] T 3fea2c89a1f2426c83ea043339dbe3c9 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:36.018503 7014 raft_consensus.cc:2454] T 3fea2c89a1f2426c83ea043339dbe3c9 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:36.018846 9296 leader_election.cc:302] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:36.019073 9823 raft_consensus.cc:2781] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:36.019359 9823 raft_consensus.cc:686] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Running, Role: LEADER
I20240503 01:59:36.019559 6883 raft_consensus.cc:2454] T 3fea2c89a1f2426c83ea043339dbe3c9 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:36.019794 9823 consensus_queue.cc:237] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:36.024039 6737 catalog_manager.cc:5484] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:36.048121 7255 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 01:59:36.049803 9343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:36.051328 6861 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:36.054960 9606 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.057593 6994 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 01:59:36.073606 9737 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.074302 9474 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.097874 7125 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:36.171994 6733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:36.179425 7014 raft_consensus.cc:1261] T 3fea2c89a1f2426c83ea043339dbe3c9 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:36.179919 6883 raft_consensus.cc:1261] T 3fea2c89a1f2426c83ea043339dbe3c9 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:36.180097 9869 consensus_queue.cc:1035] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:36.180528 9823 consensus_queue.cc:1035] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:36.186266 9907 mvcc.cc:205] Tried to move back new op lower bound from 7023417656020676608 to 7023417655381520384. Current Snapshot: MvccSnapshot[applied={T|T < 7023417656020676608}]
I20240503 01:59:36.186849 9914 tablet.cc:1722] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:36.187805 9914 tablet.cc:1620] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 01:59:36.187430 9916 tablet.cc:1722] T 3fea2c89a1f2426c83ea043339dbe3c9 P 63331f455be049c1bdd6c23c59ac1bef: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:36.188526 9916 tablet.cc:1620] T 3fea2c89a1f2426c83ea043339dbe3c9 P 63331f455be049c1bdd6c23c59ac1bef: MemRowSet was empty: no flush needed.
I20240503 01:59:36.186461 6733 catalog_manager.cc:5576] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231) does not have the latest schema for tablet 3fea2c89a1f2426c83ea043339dbe3c9 (table rebalancer_dynamic_table [id=b8e14b57de9c493d97360942327b846d]). Expected version 1 got 0
I20240503 01:59:36.190263 6737 catalog_manager.cc:5576] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943) does not have the latest schema for tablet 3fea2c89a1f2426c83ea043339dbe3c9 (table rebalancer_dynamic_table [id=b8e14b57de9c493d97360942327b846d]). Expected version 1 got 0
I20240503 01:59:36.191406 9915 tablet.cc:1722] T 3fea2c89a1f2426c83ea043339dbe3c9 P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:36.192257 9915 tablet.cc:1620] T 3fea2c89a1f2426c83ea043339dbe3c9 P 5dcc33cc621a4cbbae155f3230fb7c46: MemRowSet was empty: no flush needed.
I20240503 01:59:36.193667 6733 catalog_manager.cc:5576] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943) does not have the latest schema for tablet 3fea2c89a1f2426c83ea043339dbe3c9 (table rebalancer_dynamic_table [id=b8e14b57de9c493d97360942327b846d]). Expected version 1 got 0
I20240503 01:59:36.194423 6737 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=b8e14b57de9c493d97360942327b846d] alter complete (version 1)
I20240503 01:59:36.259666 6738 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240503 01:59:36.259894 6738 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240503 01:59:36.263377 6738 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet f56eecf7b234413aa854b77ef2094992
I20240503 01:59:36.263844 6738 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet e4c02a8a79274ca1804b127579e28e7c
I20240503 01:59:36.264142 6738 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 158f5b2278de4b4a82ee51924ee32c86
I20240503 01:59:36.264185 7255 tablet_service.cc:1514] Processing DeleteTablet for tablet f56eecf7b234413aa854b77ef2094992 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:36 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:36.264928 7125 tablet_service.cc:1514] Processing DeleteTablet for tablet f56eecf7b234413aa854b77ef2094992 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:36 UTC) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:36.265143 7254 tablet_service.cc:1514] Processing DeleteTablet for tablet 158f5b2278de4b4a82ee51924ee32c86 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:36 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:36.265283 7124 tablet_service.cc:1514] Processing DeleteTablet for tablet e4c02a8a79274ca1804b127579e28e7c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:36 UTC) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:36.265396 9953 tablet_replica.cc:330] T f56eecf7b234413aa854b77ef2094992 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:36.265744 9953 raft_consensus.cc:2229] T f56eecf7b234413aa854b77ef2094992 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:36.266039 9953 raft_consensus.cc:2258] T f56eecf7b234413aa854b77ef2094992 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:36.266204 6994 tablet_service.cc:1514] Processing DeleteTablet for tablet f56eecf7b234413aa854b77ef2094992 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:36 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:36.266295 6992 tablet_service.cc:1514] Processing DeleteTablet for tablet e4c02a8a79274ca1804b127579e28e7c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:36 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:36.266703 9343 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:36.266786 9954 tablet_replica.cc:330] T f56eecf7b234413aa854b77ef2094992 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:36.266814 6734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:36.266791 6993 tablet_service.cc:1514] Processing DeleteTablet for tablet 158f5b2278de4b4a82ee51924ee32c86 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:36 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:36.267349 9954 raft_consensus.cc:2229] T f56eecf7b234413aa854b77ef2094992 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:36.267689 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet e4c02a8a79274ca1804b127579e28e7c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:36 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:36.268319 6863 tablet_service.cc:1514] Processing DeleteTablet for tablet 158f5b2278de4b4a82ee51924ee32c86 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:36 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:36.267541 9954 raft_consensus.cc:2258] T f56eecf7b234413aa854b77ef2094992 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:36.269740 9954 ts_tablet_manager.cc:1875] T f56eecf7b234413aa854b77ef2094992 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:36.271688 7254 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 01:59:36.272449 9954 ts_tablet_manager.cc:1888] T f56eecf7b234413aa854b77ef2094992 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:36.272580 9954 log.cc:1196] T f56eecf7b234413aa854b77ef2094992 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/f56eecf7b234413aa854b77ef2094992
I20240503 01:59:36.272948 9954 ts_tablet_manager.cc:1909] T f56eecf7b234413aa854b77ef2094992 P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:36.274142 9954 tablet_replica.cc:330] T e4c02a8a79274ca1804b127579e28e7c P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:36.274305 9954 raft_consensus.cc:2229] T e4c02a8a79274ca1804b127579e28e7c P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:36.274451 9954 raft_consensus.cc:2258] T e4c02a8a79274ca1804b127579e28e7c P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:36.274608 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet f56eecf7b234413aa854b77ef2094992 (table rebalance_test_table_0 [id=f95424af8eee4c98883391a29cf46150]) successfully deleted
I20240503 01:59:36.275059 9954 ts_tablet_manager.cc:1875] T e4c02a8a79274ca1804b127579e28e7c P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:36.278395 9955 tablet_replica.cc:330] T e4c02a8a79274ca1804b127579e28e7c P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:36.278584 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 2a6014f0ba4e48efb2944d9aaf1761ab (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=72ea0c008b3e43ccab2d5b224cf8896b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:36.278995 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a6014f0ba4e48efb2944d9aaf1761ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:36.279388 9343 tablet_service.cc:1467] Processing CreateTablet for tablet 31b077072738410f8d7c139ad1b5aa9c (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=72ea0c008b3e43ccab2d5b224cf8896b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:36.279603 9343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31b077072738410f8d7c139ad1b5aa9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:36.281467 6863 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:36.281481 9821 tablet_bootstrap.cc:492] T 31b077072738410f8d7c139ad1b5aa9c P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:36.282481 9821 tablet_bootstrap.cc:654] T 31b077072738410f8d7c139ad1b5aa9c P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:36.283011 7254 tablet_service.cc:1467] Processing CreateTablet for tablet 7046186cd7f540e7b0292da5f422a4a1 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=72ea0c008b3e43ccab2d5b224cf8896b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:36.283269 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7046186cd7f540e7b0292da5f422a4a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:36.285660 9737 tablet_service.cc:1467] Processing CreateTablet for tablet 31b077072738410f8d7c139ad1b5aa9c (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=72ea0c008b3e43ccab2d5b224cf8896b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:36.286665 9737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31b077072738410f8d7c139ad1b5aa9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:36.287190 6993 tablet_service.cc:1467] Processing CreateTablet for tablet 2a6014f0ba4e48efb2944d9aaf1761ab (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=72ea0c008b3e43ccab2d5b224cf8896b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:36.287444 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a6014f0ba4e48efb2944d9aaf1761ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:36.287945 9952 tablet_replica.cc:330] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:36.288311 9958 tablet_bootstrap.cc:492] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:36.292714 9952 raft_consensus.cc:2229] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:36.293195 9952 raft_consensus.cc:2258] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:36.293962 9605 tablet_service.cc:1467] Processing CreateTablet for tablet 2a6014f0ba4e48efb2944d9aaf1761ab (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=72ea0c008b3e43ccab2d5b224cf8896b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:36.295807 9953 ts_tablet_manager.cc:1875] T f56eecf7b234413aa854b77ef2094992 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:36.296677 9952 ts_tablet_manager.cc:1875] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:36.297768 9606 tablet_service.cc:1467] Processing CreateTablet for tablet 7046186cd7f540e7b0292da5f422a4a1 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=72ea0c008b3e43ccab2d5b224cf8896b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:36.298416 9606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7046186cd7f540e7b0292da5f422a4a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:36.298820 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:36.300194 9954 ts_tablet_manager.cc:1888] T e4c02a8a79274ca1804b127579e28e7c P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:36.300271 9954 log.cc:1196] T e4c02a8a79274ca1804b127579e28e7c P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/e4c02a8a79274ca1804b127579e28e7c
I20240503 01:59:36.300523 9954 ts_tablet_manager.cc:1909] T e4c02a8a79274ca1804b127579e28e7c P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:36.300496 9953 ts_tablet_manager.cc:1888] T f56eecf7b234413aa854b77ef2094992 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:36.300590 9953 log.cc:1196] T f56eecf7b234413aa854b77ef2094992 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/f56eecf7b234413aa854b77ef2094992
I20240503 01:59:36.300938 9953 ts_tablet_manager.cc:1909] T f56eecf7b234413aa854b77ef2094992 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting consensus metadata
I20240503 01:59:36.301496 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet e4c02a8a79274ca1804b127579e28e7c (table rebalance_test_table_0 [id=f95424af8eee4c98883391a29cf46150]) successfully deleted
I20240503 01:59:36.301725 9954 tablet_replica.cc:330] T 158f5b2278de4b4a82ee51924ee32c86 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:36.301867 9954 raft_consensus.cc:2229] T 158f5b2278de4b4a82ee51924ee32c86 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:36.301992 9954 raft_consensus.cc:2258] T 158f5b2278de4b4a82ee51924ee32c86 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:36.302634 9954 ts_tablet_manager.cc:1875] T 158f5b2278de4b4a82ee51924ee32c86 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:36.306404 9914 tablet.cc:1722] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:36.306756 9914 tablet.cc:1620] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 01:59:36.310171 9954 ts_tablet_manager.cc:1888] T 158f5b2278de4b4a82ee51924ee32c86 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:36.310261 9954 log.cc:1196] T 158f5b2278de4b4a82ee51924ee32c86 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/158f5b2278de4b4a82ee51924ee32c86
I20240503 01:59:36.310534 9954 ts_tablet_manager.cc:1909] T 158f5b2278de4b4a82ee51924ee32c86 P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:36.311085 9605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a6014f0ba4e48efb2944d9aaf1761ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:36.312304 9953 tablet_replica.cc:330] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:36.312408 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet f56eecf7b234413aa854b77ef2094992 (table rebalance_test_table_0 [id=f95424af8eee4c98883391a29cf46150]) successfully deleted
I20240503 01:59:36.314167 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet 158f5b2278de4b4a82ee51924ee32c86 (table rebalance_test_table_0 [id=f95424af8eee4c98883391a29cf46150]) successfully deleted
I20240503 01:59:36.314646 9953 raft_consensus.cc:2229] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:36.314997 9953 raft_consensus.cc:2258] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:36.315150 9916 tablet.cc:1722] T 3fea2c89a1f2426c83ea043339dbe3c9 P 63331f455be049c1bdd6c23c59ac1bef: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:36.315474 9916 tablet.cc:1620] T 3fea2c89a1f2426c83ea043339dbe3c9 P 63331f455be049c1bdd6c23c59ac1bef: MemRowSet was empty: no flush needed.
I20240503 01:59:36.316759 6739 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=b8e14b57de9c493d97360942327b846d] alter complete (version 2)
I20240503 01:59:36.318212 9953 ts_tablet_manager.cc:1875] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:36.320706 9953 ts_tablet_manager.cc:1888] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:36.324944 9955 raft_consensus.cc:2229] T e4c02a8a79274ca1804b127579e28e7c P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:36.324782 9474 tablet_service.cc:1467] Processing CreateTablet for tablet 31b077072738410f8d7c139ad1b5aa9c (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=72ea0c008b3e43ccab2d5b224cf8896b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:36.325351 9955 raft_consensus.cc:2258] T e4c02a8a79274ca1804b127579e28e7c P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:36.325938 9474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31b077072738410f8d7c139ad1b5aa9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:36.326594 9955 ts_tablet_manager.cc:1875] T e4c02a8a79274ca1804b127579e28e7c P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:36.326839 9473 tablet_service.cc:1467] Processing CreateTablet for tablet 7046186cd7f540e7b0292da5f422a4a1 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=72ea0c008b3e43ccab2d5b224cf8896b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:36.327150 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7046186cd7f540e7b0292da5f422a4a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:36.324211 9953 log.cc:1196] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/e4c02a8a79274ca1804b127579e28e7c
I20240503 01:59:36.327662 9953 ts_tablet_manager.cc:1909] T e4c02a8a79274ca1804b127579e28e7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting consensus metadata
I20240503 01:59:36.330782 9915 tablet.cc:1722] T 3fea2c89a1f2426c83ea043339dbe3c9 P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:36.331118 9915 tablet.cc:1620] T 3fea2c89a1f2426c83ea043339dbe3c9 P 5dcc33cc621a4cbbae155f3230fb7c46: MemRowSet was empty: no flush needed.
I20240503 01:59:36.331555 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet e4c02a8a79274ca1804b127579e28e7c (table rebalance_test_table_0 [id=f95424af8eee4c98883391a29cf46150]) successfully deleted
I20240503 01:59:36.338315 6993 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 01:59:36.341773 9540 heartbeater.cc:503] Master 127.0.99.190:36303 was elected leader, sending a full tablet report...
I20240503 01:59:36.341974 9821 tablet_bootstrap.cc:492] T 31b077072738410f8d7c139ad1b5aa9c P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:36.342109 9821 ts_tablet_manager.cc:1367] T 31b077072738410f8d7c139ad1b5aa9c P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.061s user 0.000s sys 0.002s
I20240503 01:59:36.342713 9821 raft_consensus.cc:348] T 31b077072738410f8d7c139ad1b5aa9c P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:36.342832 9821 raft_consensus.cc:374] T 31b077072738410f8d7c139ad1b5aa9c P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:36.342864 9821 raft_consensus.cc:729] T 31b077072738410f8d7c139ad1b5aa9c P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:36.342990 9821 consensus_queue.cc:260] T 31b077072738410f8d7c139ad1b5aa9c P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:36.343235 9821 ts_tablet_manager.cc:1398] T 31b077072738410f8d7c139ad1b5aa9c P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:36.343396 9821 tablet_bootstrap.cc:492] T 2a6014f0ba4e48efb2944d9aaf1761ab P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:36.344278 9821 tablet_bootstrap.cc:654] T 2a6014f0ba4e48efb2944d9aaf1761ab P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:36.347723 9737 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.348568 9474 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.302978 9958 tablet_bootstrap.cc:654] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:36.355015 9963 tablet_bootstrap.cc:492] T 2a6014f0ba4e48efb2944d9aaf1761ab P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:36.355966 9959 tablet_bootstrap.cc:492] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:36.356139 9821 tablet_bootstrap.cc:492] T 2a6014f0ba4e48efb2944d9aaf1761ab P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:36.356315 9821 ts_tablet_manager.cc:1367] T 2a6014f0ba4e48efb2944d9aaf1761ab P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.001s
I20240503 01:59:36.356822 9821 raft_consensus.cc:348] T 2a6014f0ba4e48efb2944d9aaf1761ab P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:36.357015 9821 raft_consensus.cc:374] T 2a6014f0ba4e48efb2944d9aaf1761ab P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:36.357088 9821 raft_consensus.cc:729] T 2a6014f0ba4e48efb2944d9aaf1761ab P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:36.357259 9821 consensus_queue.cc:260] T 2a6014f0ba4e48efb2944d9aaf1761ab P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:36.359109 9959 tablet_bootstrap.cc:654] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:36.362210 9821 ts_tablet_manager.cc:1398] T 2a6014f0ba4e48efb2944d9aaf1761ab P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 01:59:36.362238 9955 ts_tablet_manager.cc:1888] T e4c02a8a79274ca1804b127579e28e7c P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:36.362337 9955 log.cc:1196] T e4c02a8a79274ca1804b127579e28e7c P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/e4c02a8a79274ca1804b127579e28e7c
I20240503 01:59:36.362789 9960 tablet_bootstrap.cc:492] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:36.363873 9963 tablet_bootstrap.cc:654] T 2a6014f0ba4e48efb2944d9aaf1761ab P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:36.364346 9955 ts_tablet_manager.cc:1909] T e4c02a8a79274ca1804b127579e28e7c P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:36.364915 9963 log.cc:824] T 2a6014f0ba4e48efb2944d9aaf1761ab P b7062fbbc2b54fff9e99ec4ea5562c76: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:36.355023 9952 ts_tablet_manager.cc:1888] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:36.365376 9952 log.cc:1196] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/f56eecf7b234413aa854b77ef2094992
I20240503 01:59:36.366298 9952 ts_tablet_manager.cc:1909] T f56eecf7b234413aa854b77ef2094992 P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:36.366377 9960 tablet_bootstrap.cc:654] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:36.365931 9958 tablet_bootstrap.cc:492] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:36.366644 9958 ts_tablet_manager.cc:1367] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.078s user 0.002s sys 0.000s
I20240503 01:59:36.367152 9958 raft_consensus.cc:348] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:36.367277 9958 raft_consensus.cc:374] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:36.367311 9958 raft_consensus.cc:729] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:36.367419 9958 consensus_queue.cc:260] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:36.368598 7124 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:36.369537 9958 ts_tablet_manager.cc:1398] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:36.371479 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet f56eecf7b234413aa854b77ef2094992 (table rebalance_test_table_0 [id=f95424af8eee4c98883391a29cf46150]) successfully deleted
I20240503 01:59:36.371677 9952 tablet_replica.cc:330] T 158f5b2278de4b4a82ee51924ee32c86 P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:36.371876 9952 raft_consensus.cc:2229] T 158f5b2278de4b4a82ee51924ee32c86 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:36.372246 9952 raft_consensus.cc:2258] T 158f5b2278de4b4a82ee51924ee32c86 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:36.372888 9952 ts_tablet_manager.cc:1875] T 158f5b2278de4b4a82ee51924ee32c86 P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:36.375334 9952 ts_tablet_manager.cc:1888] T 158f5b2278de4b4a82ee51924ee32c86 P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:36.375474 9952 log.cc:1196] T 158f5b2278de4b4a82ee51924ee32c86 P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/158f5b2278de4b4a82ee51924ee32c86
I20240503 01:59:36.375785 9952 ts_tablet_manager.cc:1909] T 158f5b2278de4b4a82ee51924ee32c86 P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:36.375778 9606 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.366999 9959 tablet_bootstrap.cc:492] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:36.376046 9959 ts_tablet_manager.cc:1367] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240503 01:59:36.376569 9959 raft_consensus.cc:348] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:36.376695 9959 raft_consensus.cc:374] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:36.376725 9959 raft_consensus.cc:729] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:36.376843 9959 consensus_queue.cc:260] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:36.377108 9959 ts_tablet_manager.cc:1398] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:36.377977 9955 tablet_replica.cc:330] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:36.378137 9955 raft_consensus.cc:2229] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:36.378106 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet e4c02a8a79274ca1804b127579e28e7c (table rebalance_test_table_0 [id=f95424af8eee4c98883391a29cf46150]) successfully deleted
I20240503 01:59:36.378389 9955 raft_consensus.cc:2258] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:36.378683 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet 158f5b2278de4b4a82ee51924ee32c86 (table rebalance_test_table_0 [id=f95424af8eee4c98883391a29cf46150]) successfully deleted
I20240503 01:59:36.379033 9955 ts_tablet_manager.cc:1875] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:36.382165 9913 raft_consensus.cc:482] T 31b077072738410f8d7c139ad1b5aa9c P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:36.382167 9960 log.cc:824] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:36.382272 9913 raft_consensus.cc:504] T 31b077072738410f8d7c139ad1b5aa9c P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:36.382899 9913 leader_election.cc:288] T 31b077072738410f8d7c139ad1b5aa9c P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:36.391158 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b077072738410f8d7c139ad1b5aa9c" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
W20240503 01:59:36.393021 9297 leader_election.cc:341] T 31b077072738410f8d7c139ad1b5aa9c P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): Illegal state: must be running to vote when last-logged opid is not known
I20240503 01:59:36.394354 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b077072738410f8d7c139ad1b5aa9c" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
W20240503 01:59:36.396538 9296 leader_election.cc:341] T 31b077072738410f8d7c139ad1b5aa9c P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Illegal state: must be running to vote when last-logged opid is not known
I20240503 01:59:36.396646 9296 leader_election.cc:302] T 31b077072738410f8d7c139ad1b5aa9c P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8; no voters: bb5364d730cb4489a0bfc6c270b7450a, db1a0000aaa346f79c0aa4b6c5105191
I20240503 01:59:36.396932 9913 raft_consensus.cc:2726] T 31b077072738410f8d7c139ad1b5aa9c P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 01:59:36.397869 9955 ts_tablet_manager.cc:1888] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:36.397958 9955 log.cc:1196] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/158f5b2278de4b4a82ee51924ee32c86
I20240503 01:59:36.398211 9955 ts_tablet_manager.cc:1909] T 158f5b2278de4b4a82ee51924ee32c86 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:36.399417 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 158f5b2278de4b4a82ee51924ee32c86 (table rebalance_test_table_0 [id=f95424af8eee4c98883391a29cf46150]) successfully deleted
I20240503 01:59:36.400127 9963 tablet_bootstrap.cc:492] T 2a6014f0ba4e48efb2944d9aaf1761ab P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:36.400338 9963 ts_tablet_manager.cc:1367] T 2a6014f0ba4e48efb2944d9aaf1761ab P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.000s
I20240503 01:59:36.403892 9963 raft_consensus.cc:348] T 2a6014f0ba4e48efb2944d9aaf1761ab P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:36.404124 9963 raft_consensus.cc:374] T 2a6014f0ba4e48efb2944d9aaf1761ab P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:36.404186 9963 raft_consensus.cc:729] T 2a6014f0ba4e48efb2944d9aaf1761ab P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:36.404650 9963 consensus_queue.cc:260] T 2a6014f0ba4e48efb2944d9aaf1761ab P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:36.405912 9963 ts_tablet_manager.cc:1398] T 2a6014f0ba4e48efb2944d9aaf1761ab P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20240503 01:59:36.406311 9963 tablet_bootstrap.cc:492] T 7046186cd7f540e7b0292da5f422a4a1 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:36.406314 9672 heartbeater.cc:503] Master 127.0.99.190:36303 was elected leader, sending a full tablet report...
I20240503 01:59:36.407562 9963 tablet_bootstrap.cc:654] T 7046186cd7f540e7b0292da5f422a4a1 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:36.411005 9965 tablet_bootstrap.cc:492] T 31b077072738410f8d7c139ad1b5aa9c P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:36.413648 9963 tablet_bootstrap.cc:492] T 7046186cd7f540e7b0292da5f422a4a1 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:36.413877 9963 ts_tablet_manager.cc:1367] T 7046186cd7f540e7b0292da5f422a4a1 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s
I20240503 01:59:36.414710 9963 raft_consensus.cc:348] T 7046186cd7f540e7b0292da5f422a4a1 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:36.414925 9963 raft_consensus.cc:374] T 7046186cd7f540e7b0292da5f422a4a1 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:36.415000 9963 raft_consensus.cc:729] T 7046186cd7f540e7b0292da5f422a4a1 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:36.415220 9963 consensus_queue.cc:260] T 7046186cd7f540e7b0292da5f422a4a1 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:36.415649 9960 tablet_bootstrap.cc:492] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:36.415879 9960 ts_tablet_manager.cc:1367] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.002s
I20240503 01:59:36.416250 9963 ts_tablet_manager.cc:1398] T 7046186cd7f540e7b0292da5f422a4a1 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:36.419381 9960 raft_consensus.cc:348] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:36.419631 9960 raft_consensus.cc:374] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:36.419806 9960 raft_consensus.cc:729] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:36.420274 9960 consensus_queue.cc:260] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:36.413733 9965 tablet_bootstrap.cc:654] T 31b077072738410f8d7c139ad1b5aa9c P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:36.421191 9803 heartbeater.cc:503] Master 127.0.99.190:36303 was elected leader, sending a full tablet report...
I20240503 01:59:36.421996 9965 log.cc:824] T 31b077072738410f8d7c139ad1b5aa9c P bb5364d730cb4489a0bfc6c270b7450a: Log is configured to *not* fsync() on all Append() calls
I20240503 01:59:36.422302 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:36.422920 9960 ts_tablet_manager.cc:1398] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20240503 01:59:36.424135 9965 tablet_bootstrap.cc:492] T 31b077072738410f8d7c139ad1b5aa9c P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:36.424394 9965 ts_tablet_manager.cc:1367] T 31b077072738410f8d7c139ad1b5aa9c P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20240503 01:59:36.428457 9965 raft_consensus.cc:348] T 31b077072738410f8d7c139ad1b5aa9c P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:36.428701 9965 raft_consensus.cc:374] T 31b077072738410f8d7c139ad1b5aa9c P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:36.428900 9914 tablet.cc:1722] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:36.431483 9916 tablet.cc:1722] T 3fea2c89a1f2426c83ea043339dbe3c9 P 63331f455be049c1bdd6c23c59ac1bef: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:36.431918 9965 raft_consensus.cc:729] T 31b077072738410f8d7c139ad1b5aa9c P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:36.432718 9965 consensus_queue.cc:260] T 31b077072738410f8d7c139ad1b5aa9c P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:36.433111 9915 tablet.cc:1722] T 3fea2c89a1f2426c83ea043339dbe3c9 P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:36.433904 9965 ts_tablet_manager.cc:1398] T 31b077072738410f8d7c139ad1b5aa9c P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20240503 01:59:36.434329 9965 tablet_bootstrap.cc:492] T 7046186cd7f540e7b0292da5f422a4a1 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:36.435487 9965 tablet_bootstrap.cc:654] T 7046186cd7f540e7b0292da5f422a4a1 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:36.437927 6739 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=b8e14b57de9c493d97360942327b846d] alter complete (version 3)
I20240503 01:59:36.443467 9965 tablet_bootstrap.cc:492] T 7046186cd7f540e7b0292da5f422a4a1 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:36.443588 9965 ts_tablet_manager.cc:1367] T 7046186cd7f540e7b0292da5f422a4a1 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 01:59:36.444200 9965 raft_consensus.cc:348] T 7046186cd7f540e7b0292da5f422a4a1 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:36.444391 9965 raft_consensus.cc:374] T 7046186cd7f540e7b0292da5f422a4a1 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:36.444447 9965 raft_consensus.cc:729] T 7046186cd7f540e7b0292da5f422a4a1 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:36.444583 9965 consensus_queue.cc:260] T 7046186cd7f540e7b0292da5f422a4a1 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:36.444871 9965 ts_tablet_manager.cc:1398] T 7046186cd7f540e7b0292da5f422a4a1 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:36.453054 9913 raft_consensus.cc:482] T 2a6014f0ba4e48efb2944d9aaf1761ab P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:36.453208 9913 raft_consensus.cc:504] T 2a6014f0ba4e48efb2944d9aaf1761ab P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:36.453709 9913 leader_election.cc:288] T 2a6014f0ba4e48efb2944d9aaf1761ab P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:36.454886 7013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6014f0ba4e48efb2944d9aaf1761ab" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:36.455046 7013 raft_consensus.cc:2454] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:36.455574 9296 leader_election.cc:302] T 2a6014f0ba4e48efb2944d9aaf1761ab P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:36.455765 9913 raft_consensus.cc:2781] T 2a6014f0ba4e48efb2944d9aaf1761ab P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:36.455871 9913 raft_consensus.cc:482] T 2a6014f0ba4e48efb2944d9aaf1761ab P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:36.455924 9913 raft_consensus.cc:3037] T 2a6014f0ba4e48efb2944d9aaf1761ab P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:36.456838 9913 raft_consensus.cc:504] T 2a6014f0ba4e48efb2944d9aaf1761ab P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:36.457243 9913 leader_election.cc:288] T 2a6014f0ba4e48efb2944d9aaf1761ab P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:36.458328 7013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6014f0ba4e48efb2944d9aaf1761ab" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:36.458460 7013 raft_consensus.cc:3037] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:36.458494 9626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6014f0ba4e48efb2944d9aaf1761ab" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:36.458494 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a6014f0ba4e48efb2944d9aaf1761ab" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:36.459023 9626 raft_consensus.cc:2454] T 2a6014f0ba4e48efb2944d9aaf1761ab P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:36.459558 7013 raft_consensus.cc:2454] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:36.460054 9296 leader_election.cc:302] T 2a6014f0ba4e48efb2944d9aaf1761ab P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 63331f455be049c1bdd6c23c59ac1bef; no voters: b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 01:59:36.460273 9913 raft_consensus.cc:2781] T 2a6014f0ba4e48efb2944d9aaf1761ab P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:36.460356 9913 raft_consensus.cc:686] T 2a6014f0ba4e48efb2944d9aaf1761ab P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Running, Role: LEADER
I20240503 01:59:36.460498 9913 consensus_queue.cc:237] T 2a6014f0ba4e48efb2944d9aaf1761ab P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:36.462302 6734 catalog_manager.cc:5484] T 2a6014f0ba4e48efb2944d9aaf1761ab P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:36.492954 8933 raft_consensus.cc:482] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:36.493141 8933 raft_consensus.cc:504] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:36.493773 8933 leader_election.cc:288] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:36.498015 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7046186cd7f540e7b0292da5f422a4a1" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:36.498360 9494 raft_consensus.cc:2454] T 7046186cd7f540e7b0292da5f422a4a1 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:36.498890 7209 leader_election.cc:302] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:36.499140 8933 raft_consensus.cc:2781] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:36.499198 8933 raft_consensus.cc:482] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:36.499236 8933 raft_consensus.cc:3037] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:36.500380 8933 raft_consensus.cc:504] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:36.500428 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7046186cd7f540e7b0292da5f422a4a1" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:36.500578 9625 raft_consensus.cc:2454] T 7046186cd7f540e7b0292da5f422a4a1 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:36.500720 8933 leader_election.cc:288] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:36.501206 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7046186cd7f540e7b0292da5f422a4a1" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:36.501377 9625 raft_consensus.cc:3037] T 7046186cd7f540e7b0292da5f422a4a1 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:36.501433 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7046186cd7f540e7b0292da5f422a4a1" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:36.501610 9494 raft_consensus.cc:3037] T 7046186cd7f540e7b0292da5f422a4a1 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:36.502705 9625 raft_consensus.cc:2454] T 7046186cd7f540e7b0292da5f422a4a1 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:36.502705 9494 raft_consensus.cc:2454] T 7046186cd7f540e7b0292da5f422a4a1 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
W20240503 01:59:36.502964 9804 tablet.cc:2369] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:36.503026 7211 leader_election.cc:302] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:36.503275 9547 raft_consensus.cc:2781] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:36.503532 9547 raft_consensus.cc:686] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 01:59:36.503690 9547 consensus_queue.cc:237] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:36.505831 6734 catalog_manager.cc:5484] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
W20240503 01:59:36.536867 9673 tablet.cc:2369] T 2a6014f0ba4e48efb2944d9aaf1761ab P b7062fbbc2b54fff9e99ec4ea5562c76: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:36.539047 6738 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:36.539219 6738 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:36.541775 6738 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 3fea2c89a1f2426c83ea043339dbe3c9
I20240503 01:59:36.542470 9342 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fea2c89a1f2426c83ea043339dbe3c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:36 UTC) from {username='slave'} at 127.0.0.1:36040
I20240503 01:59:36.542416 6863 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fea2c89a1f2426c83ea043339dbe3c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:36 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:36.543274 9955 tablet_replica.cc:330] T 3fea2c89a1f2426c83ea043339dbe3c9 P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:36.543434 9955 raft_consensus.cc:2229] T 3fea2c89a1f2426c83ea043339dbe3c9 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:36.543572 9955 raft_consensus.cc:2258] T 3fea2c89a1f2426c83ea043339dbe3c9 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:36.544226 10023 tablet_replica.cc:330] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8: stopping tablet replica
I20240503 01:59:36.544623 10023 raft_consensus.cc:2229] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:36.545092 10023 raft_consensus.cc:2258] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:36.546254 9955 ts_tablet_manager.cc:1875] T 3fea2c89a1f2426c83ea043339dbe3c9 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:36.547397 10023 ts_tablet_manager.cc:1875] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:36.549212 9955 ts_tablet_manager.cc:1888] T 3fea2c89a1f2426c83ea043339dbe3c9 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:36.551791 10023 ts_tablet_manager.cc:1888] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:36.551934 10023 log.cc:1196] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/3fea2c89a1f2426c83ea043339dbe3c9
I20240503 01:59:36.552284 10023 ts_tablet_manager.cc:1909] T 3fea2c89a1f2426c83ea043339dbe3c9 P 12e8c683db054df3b75c6285b98a5ad8: Deleting consensus metadata
I20240503 01:59:36.553534 6727 catalog_manager.cc:4830] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): tablet 3fea2c89a1f2426c83ea043339dbe3c9 (table rebalancer_dynamic_table_new_name [id=b8e14b57de9c493d97360942327b846d]) successfully deleted
I20240503 01:59:36.554890 9955 log.cc:1196] T 3fea2c89a1f2426c83ea043339dbe3c9 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/3fea2c89a1f2426c83ea043339dbe3c9
I20240503 01:59:36.555238 9955 ts_tablet_manager.cc:1909] T 3fea2c89a1f2426c83ea043339dbe3c9 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:36.556175 6993 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fea2c89a1f2426c83ea043339dbe3c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:36 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:36.556576 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 3fea2c89a1f2426c83ea043339dbe3c9 (table rebalancer_dynamic_table_new_name [id=b8e14b57de9c493d97360942327b846d]) successfully deleted
I20240503 01:59:36.556702 9954 tablet_replica.cc:330] T 3fea2c89a1f2426c83ea043339dbe3c9 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:36.556852 9954 raft_consensus.cc:2229] T 3fea2c89a1f2426c83ea043339dbe3c9 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:36.556958 9954 raft_consensus.cc:2258] T 3fea2c89a1f2426c83ea043339dbe3c9 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:36.557693 9954 ts_tablet_manager.cc:1875] T 3fea2c89a1f2426c83ea043339dbe3c9 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:36.560110 9954 ts_tablet_manager.cc:1888] T 3fea2c89a1f2426c83ea043339dbe3c9 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:36.560209 9954 log.cc:1196] T 3fea2c89a1f2426c83ea043339dbe3c9 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/3fea2c89a1f2426c83ea043339dbe3c9
I20240503 01:59:36.560472 9954 ts_tablet_manager.cc:1909] T 3fea2c89a1f2426c83ea043339dbe3c9 P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:36.561295 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet 3fea2c89a1f2426c83ea043339dbe3c9 (table rebalancer_dynamic_table_new_name [id=b8e14b57de9c493d97360942327b846d]) successfully deleted
W20240503 01:59:36.578294 9541 tablet.cc:2369] T 31b077072738410f8d7c139ad1b5aa9c P bb5364d730cb4489a0bfc6c270b7450a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:36.657459 6733 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:36.663128 9737 tablet_service.cc:1467] Processing CreateTablet for tablet 574a8dbc94774bd3b2129049d52892bc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=03ad9337878a45fe978b426b519046d7]), partition=
I20240503 01:59:36.663429 9737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 574a8dbc94774bd3b2129049d52892bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:36.663400 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 574a8dbc94774bd3b2129049d52892bc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=03ad9337878a45fe978b426b519046d7]), partition=
I20240503 01:59:36.663501 7254 tablet_service.cc:1467] Processing CreateTablet for tablet 574a8dbc94774bd3b2129049d52892bc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=03ad9337878a45fe978b426b519046d7]), partition=
I20240503 01:59:36.663654 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 574a8dbc94774bd3b2129049d52892bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:36.663714 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 574a8dbc94774bd3b2129049d52892bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:36.665570 9960 tablet_bootstrap.cc:492] T 574a8dbc94774bd3b2129049d52892bc P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:36.665760 9958 tablet_bootstrap.cc:492] T 574a8dbc94774bd3b2129049d52892bc P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:36.666661 9958 tablet_bootstrap.cc:654] T 574a8dbc94774bd3b2129049d52892bc P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:36.666989 9960 tablet_bootstrap.cc:654] T 574a8dbc94774bd3b2129049d52892bc P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:36.667198 9821 tablet_bootstrap.cc:492] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:36.667770 9958 tablet_bootstrap.cc:492] T 574a8dbc94774bd3b2129049d52892bc P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:36.667847 9958 ts_tablet_manager.cc:1367] T 574a8dbc94774bd3b2129049d52892bc P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:36.668051 9960 tablet_bootstrap.cc:492] T 574a8dbc94774bd3b2129049d52892bc P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:36.668118 9960 ts_tablet_manager.cc:1367] T 574a8dbc94774bd3b2129049d52892bc P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:59:36.668138 9821 tablet_bootstrap.cc:654] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:36.668569 9960 raft_consensus.cc:348] T 574a8dbc94774bd3b2129049d52892bc P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:36.668569 9958 raft_consensus.cc:348] T 574a8dbc94774bd3b2129049d52892bc P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:36.668681 9960 raft_consensus.cc:374] T 574a8dbc94774bd3b2129049d52892bc P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:36.668706 9958 raft_consensus.cc:374] T 574a8dbc94774bd3b2129049d52892bc P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:36.668715 9960 raft_consensus.cc:729] T 574a8dbc94774bd3b2129049d52892bc P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:36.668743 9958 raft_consensus.cc:729] T 574a8dbc94774bd3b2129049d52892bc P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:36.668852 9960 consensus_queue.cc:260] T 574a8dbc94774bd3b2129049d52892bc P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:36.668870 9958 consensus_queue.cc:260] T 574a8dbc94774bd3b2129049d52892bc P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:36.669128 9960 ts_tablet_manager.cc:1398] T 574a8dbc94774bd3b2129049d52892bc P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 01:59:36.669278 9958 ts_tablet_manager.cc:1398] T 574a8dbc94774bd3b2129049d52892bc P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 01:59:36.669533 9821 tablet_bootstrap.cc:492] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:36.669625 9821 ts_tablet_manager.cc:1367] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:36.670068 9821 raft_consensus.cc:348] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:36.670174 9821 raft_consensus.cc:374] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:36.670215 9821 raft_consensus.cc:729] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:36.670327 9821 consensus_queue.cc:260] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:36.670653 9821 ts_tablet_manager.cc:1398] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:36.715916 9913 raft_consensus.cc:482] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:36.716092 9913 raft_consensus.cc:504] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:36.716630 9913 leader_election.cc:288] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:36.717116 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574a8dbc94774bd3b2129049d52892bc" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:36.717509 9757 raft_consensus.cc:2454] T 574a8dbc94774bd3b2129049d52892bc P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:36.717976 9297 leader_election.cc:302] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:36.718199 9913 raft_consensus.cc:2781] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:36.718271 9913 raft_consensus.cc:482] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:36.718322 9913 raft_consensus.cc:3037] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:36.719462 9913 raft_consensus.cc:504] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:36.719882 9913 leader_election.cc:288] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Requested vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:36.720503 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574a8dbc94774bd3b2129049d52892bc" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:36.720680 9757 raft_consensus.cc:3037] T 574a8dbc94774bd3b2129049d52892bc P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:36.721570 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574a8dbc94774bd3b2129049d52892bc" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:36.721570 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574a8dbc94774bd3b2129049d52892bc" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:36.721727 7274 raft_consensus.cc:3037] T 574a8dbc94774bd3b2129049d52892bc P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:36.721868 9757 raft_consensus.cc:2454] T 574a8dbc94774bd3b2129049d52892bc P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:36.722213 9297 leader_election.cc:302] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:36.722492 9913 raft_consensus.cc:2781] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:36.722642 9913 raft_consensus.cc:686] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Running, Role: LEADER
I20240503 01:59:36.722818 7274 raft_consensus.cc:2454] T 574a8dbc94774bd3b2129049d52892bc P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:36.722807 9913 consensus_queue.cc:237] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:36.724577 6740 catalog_manager.cc:5484] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:36.755523 9973 raft_consensus.cc:482] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:36.755692 9973 raft_consensus.cc:504] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:36.757002 9973 leader_election.cc:288] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:36.762120 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b077072738410f8d7c139ad1b5aa9c" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:36.762339 9494 raft_consensus.cc:2454] T 31b077072738410f8d7c139ad1b5aa9c P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 01:59:36.762836 9690 leader_election.cc:302] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:36.763121 9973 raft_consensus.cc:2781] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:36.763185 9973 raft_consensus.cc:482] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:36.763221 9973 raft_consensus.cc:3037] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:36.763474 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b077072738410f8d7c139ad1b5aa9c" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:36.763711 9363 raft_consensus.cc:2454] T 31b077072738410f8d7c139ad1b5aa9c P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 01:59:36.764616 9973 raft_consensus.cc:504] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:36.765090 9973 leader_election.cc:288] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Requested vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:36.765516 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b077072738410f8d7c139ad1b5aa9c" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:36.765655 9494 raft_consensus.cc:3037] T 31b077072738410f8d7c139ad1b5aa9c P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:36.766664 9494 raft_consensus.cc:2454] T 31b077072738410f8d7c139ad1b5aa9c P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:36.766675 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b077072738410f8d7c139ad1b5aa9c" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:36.766813 9363 raft_consensus.cc:3037] T 31b077072738410f8d7c139ad1b5aa9c P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:36.767174 9690 leader_election.cc:302] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:36.767562 9973 raft_consensus.cc:2781] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:36.767645 6740 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240503 01:59:36.767778 6740 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240503 01:59:36.767787 9973 raft_consensus.cc:686] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Becoming Leader. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Running, Role: LEADER
I20240503 01:59:36.767860 9363 raft_consensus.cc:2454] T 31b077072738410f8d7c139ad1b5aa9c P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:36.768157 9973 consensus_queue.cc:237] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:36.770304 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet d9896016473544858f54bc801b689a05
I20240503 01:59:36.770691 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet b510e069ae4c4e86ab35c6f43f9bbdfe
I20240503 01:59:36.770917 7254 tablet_service.cc:1514] Processing DeleteTablet for tablet d9896016473544858f54bc801b689a05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:36 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:36.770974 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 5de30ce4152042bba22b892be1e61712
I20240503 01:59:36.771067 6733 catalog_manager.cc:5484] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: term changed from 0 to 1, leader changed from <none> to db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:36.771350 9952 tablet_replica.cc:330] T d9896016473544858f54bc801b689a05 P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:36.771468 7254 tablet_service.cc:1514] Processing DeleteTablet for tablet b510e069ae4c4e86ab35c6f43f9bbdfe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:36 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:36.771513 9952 raft_consensus.cc:2229] T d9896016473544858f54bc801b689a05 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:36.771659 9952 raft_consensus.cc:2258] T d9896016473544858f54bc801b689a05 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:36.771944 7254 tablet_service.cc:1514] Processing DeleteTablet for tablet 5de30ce4152042bba22b892be1e61712 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:36 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:36.772131 7124 tablet_service.cc:1514] Processing DeleteTablet for tablet b510e069ae4c4e86ab35c6f43f9bbdfe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:36 UTC) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:36.772264 9952 ts_tablet_manager.cc:1875] T d9896016473544858f54bc801b689a05 P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:36.772311 9953 tablet_replica.cc:330] T b510e069ae4c4e86ab35c6f43f9bbdfe P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:36.772435 9953 raft_consensus.cc:2229] T b510e069ae4c4e86ab35c6f43f9bbdfe P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:36.772574 9953 raft_consensus.cc:2258] T b510e069ae4c4e86ab35c6f43f9bbdfe P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:36.772625 6993 tablet_service.cc:1514] Processing DeleteTablet for tablet d9896016473544858f54bc801b689a05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:36 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:36.772806 9954 tablet_replica.cc:330] T d9896016473544858f54bc801b689a05 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:36.772926 9954 raft_consensus.cc:2229] T d9896016473544858f54bc801b689a05 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:36.773069 9954 raft_consensus.cc:2258] T d9896016473544858f54bc801b689a05 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:36.773079 9953 ts_tablet_manager.cc:1875] T b510e069ae4c4e86ab35c6f43f9bbdfe P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:36.773543 9954 ts_tablet_manager.cc:1875] T d9896016473544858f54bc801b689a05 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:36.774458 9952 ts_tablet_manager.cc:1888] T d9896016473544858f54bc801b689a05 P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:36.774565 6994 tablet_service.cc:1514] Processing DeleteTablet for tablet 5de30ce4152042bba22b892be1e61712 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:36 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:36.774747 6992 tablet_service.cc:1514] Processing DeleteTablet for tablet b510e069ae4c4e86ab35c6f43f9bbdfe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:36 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:36.775449 9953 ts_tablet_manager.cc:1888] T b510e069ae4c4e86ab35c6f43f9bbdfe P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:36.775485 6863 tablet_service.cc:1514] Processing DeleteTablet for tablet d9896016473544858f54bc801b689a05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:36 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:36.775569 9953 log.cc:1196] T b510e069ae4c4e86ab35c6f43f9bbdfe P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/b510e069ae4c4e86ab35c6f43f9bbdfe
I20240503 01:59:36.775672 9955 tablet_replica.cc:330] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:36.775786 9954 ts_tablet_manager.cc:1888] T d9896016473544858f54bc801b689a05 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:36.775843 9953 ts_tablet_manager.cc:1909] T b510e069ae4c4e86ab35c6f43f9bbdfe P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting consensus metadata
I20240503 01:59:36.775852 9954 log.cc:1196] T d9896016473544858f54bc801b689a05 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/d9896016473544858f54bc801b689a05
I20240503 01:59:36.775847 9955 raft_consensus.cc:2229] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:36.776067 9955 raft_consensus.cc:2258] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:36.776094 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet 5de30ce4152042bba22b892be1e61712 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:36 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:36.776561 9954 ts_tablet_manager.cc:1909] T d9896016473544858f54bc801b689a05 P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:36.776577 9955 ts_tablet_manager.cc:1875] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:36.776557 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet b510e069ae4c4e86ab35c6f43f9bbdfe (table rebalance_test_table_2 [id=ca54251151134e5998e64c31c165357e]) successfully deleted
I20240503 01:59:36.778652 9955 ts_tablet_manager.cc:1888] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:36.778726 9955 log.cc:1196] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/d9896016473544858f54bc801b689a05
I20240503 01:59:36.778688 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet d9896016473544858f54bc801b689a05 (table rebalance_test_table_2 [id=ca54251151134e5998e64c31c165357e]) successfully deleted
I20240503 01:59:36.778999 9955 ts_tablet_manager.cc:1909] T d9896016473544858f54bc801b689a05 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:36.779129 9954 tablet_replica.cc:330] T 5de30ce4152042bba22b892be1e61712 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:36.779430 9954 raft_consensus.cc:2229] T 5de30ce4152042bba22b892be1e61712 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:36.779693 9954 raft_consensus.cc:2258] T 5de30ce4152042bba22b892be1e61712 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:36.779759 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet d9896016473544858f54bc801b689a05 (table rebalance_test_table_2 [id=ca54251151134e5998e64c31c165357e]) successfully deleted
I20240503 01:59:36.780040 9955 tablet_replica.cc:330] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:36.780064 9952 log.cc:1196] T d9896016473544858f54bc801b689a05 P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/d9896016473544858f54bc801b689a05
I20240503 01:59:36.780354 9955 raft_consensus.cc:2229] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:36.780429 9952 ts_tablet_manager.cc:1909] T d9896016473544858f54bc801b689a05 P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:36.780712 9954 ts_tablet_manager.cc:1875] T 5de30ce4152042bba22b892be1e61712 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:36.780774 9955 raft_consensus.cc:2258] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:36.781268 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet d9896016473544858f54bc801b689a05 (table rebalance_test_table_2 [id=ca54251151134e5998e64c31c165357e]) successfully deleted
I20240503 01:59:36.781603 9955 ts_tablet_manager.cc:1875] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:36.781950 9952 tablet_replica.cc:330] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:36.782209 9952 raft_consensus.cc:2229] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:36.782461 9952 raft_consensus.cc:2258] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:36.783115 9952 ts_tablet_manager.cc:1875] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:36.784287 9955 ts_tablet_manager.cc:1888] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:36.784492 9955 log.cc:1196] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/5de30ce4152042bba22b892be1e61712
I20240503 01:59:36.784837 9955 ts_tablet_manager.cc:1909] T 5de30ce4152042bba22b892be1e61712 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:36.785748 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 5de30ce4152042bba22b892be1e61712 (table rebalance_test_table_2 [id=ca54251151134e5998e64c31c165357e]) successfully deleted
I20240503 01:59:36.787258 9954 ts_tablet_manager.cc:1888] T 5de30ce4152042bba22b892be1e61712 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:36.787389 9954 log.cc:1196] T 5de30ce4152042bba22b892be1e61712 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/5de30ce4152042bba22b892be1e61712
I20240503 01:59:36.787708 9954 ts_tablet_manager.cc:1909] T 5de30ce4152042bba22b892be1e61712 P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:36.788455 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet 5de30ce4152042bba22b892be1e61712 (table rebalance_test_table_2 [id=ca54251151134e5998e64c31c165357e]) successfully deleted
I20240503 01:59:36.788518 9954 tablet_replica.cc:330] T b510e069ae4c4e86ab35c6f43f9bbdfe P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:36.788951 9954 raft_consensus.cc:2229] T b510e069ae4c4e86ab35c6f43f9bbdfe P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:36.789259 9954 raft_consensus.cc:2258] T b510e069ae4c4e86ab35c6f43f9bbdfe P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:36.789373 9952 ts_tablet_manager.cc:1888] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:36.789512 9952 log.cc:1196] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/b510e069ae4c4e86ab35c6f43f9bbdfe
I20240503 01:59:36.789848 9952 ts_tablet_manager.cc:1909] T b510e069ae4c4e86ab35c6f43f9bbdfe P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:36.790181 9954 ts_tablet_manager.cc:1875] T b510e069ae4c4e86ab35c6f43f9bbdfe P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:36.790643 9952 tablet_replica.cc:330] T 5de30ce4152042bba22b892be1e61712 P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:36.790819 9952 raft_consensus.cc:2229] T 5de30ce4152042bba22b892be1e61712 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:36.790763 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet b510e069ae4c4e86ab35c6f43f9bbdfe (table rebalance_test_table_2 [id=ca54251151134e5998e64c31c165357e]) successfully deleted
I20240503 01:59:36.790949 9952 raft_consensus.cc:2258] T 5de30ce4152042bba22b892be1e61712 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:36.791517 9952 ts_tablet_manager.cc:1875] T 5de30ce4152042bba22b892be1e61712 P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:36.792776 9954 ts_tablet_manager.cc:1888] T b510e069ae4c4e86ab35c6f43f9bbdfe P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:36.792870 9954 log.cc:1196] T b510e069ae4c4e86ab35c6f43f9bbdfe P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/b510e069ae4c4e86ab35c6f43f9bbdfe
I20240503 01:59:36.793124 9954 ts_tablet_manager.cc:1909] T b510e069ae4c4e86ab35c6f43f9bbdfe P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:36.793855 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet b510e069ae4c4e86ab35c6f43f9bbdfe (table rebalance_test_table_2 [id=ca54251151134e5998e64c31c165357e]) successfully deleted
I20240503 01:59:36.795044 9952 ts_tablet_manager.cc:1888] T 5de30ce4152042bba22b892be1e61712 P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:36.795181 9952 log.cc:1196] T 5de30ce4152042bba22b892be1e61712 P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/5de30ce4152042bba22b892be1e61712
I20240503 01:59:36.795526 9952 ts_tablet_manager.cc:1909] T 5de30ce4152042bba22b892be1e61712 P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:36.796833 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet 5de30ce4152042bba22b892be1e61712 (table rebalance_test_table_2 [id=ca54251151134e5998e64c31c165357e]) successfully deleted
I20240503 01:59:36.803069 9342 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:36.814841 6992 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 01:59:36.815781 9474 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.817324 6861 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:36.818089 7124 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:36.819962 7254 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 01:59:36.826035 9737 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:36.835256 6733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:36.836673 9606 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:36.840945 9757 raft_consensus.cc:1261] T 574a8dbc94774bd3b2129049d52892bc P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:36.841775 9913 consensus_queue.cc:1035] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:36.843698 7275 raft_consensus.cc:1261] T 574a8dbc94774bd3b2129049d52892bc P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:36.844300 9913 consensus_queue.cc:1035] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:36.850979 6733 catalog_manager.cc:5576] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053) does not have the latest schema for tablet 574a8dbc94774bd3b2129049d52892bc (table rebalancer_dynamic_table [id=03ad9337878a45fe978b426b519046d7]). Expected version 1 got 0
I20240503 01:59:36.854051 9914 tablet.cc:1722] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:36.854339 9625 raft_consensus.cc:3037] T 2a6014f0ba4e48efb2944d9aaf1761ab P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:36.854410 9914 tablet.cc:1620] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 01:59:36.856069 9869 consensus_queue.cc:1035] T 2a6014f0ba4e48efb2944d9aaf1761ab P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:36.856374 6733 catalog_manager.cc:5576] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271) does not have the latest schema for tablet 574a8dbc94774bd3b2129049d52892bc (table rebalancer_dynamic_table [id=03ad9337878a45fe978b426b519046d7]). Expected version 1 got 0
I20240503 01:59:36.862567 6740 catalog_manager.cc:5576] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271) does not have the latest schema for tablet 574a8dbc94774bd3b2129049d52892bc (table rebalancer_dynamic_table [id=03ad9337878a45fe978b426b519046d7]). Expected version 1 got 0
I20240503 01:59:36.865114 9913 consensus_queue.cc:1035] T 2a6014f0ba4e48efb2944d9aaf1761ab P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:36.867126 6733 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=03ad9337878a45fe978b426b519046d7] alter complete (version 1)
I20240503 01:59:36.879674 10048 mvcc.cc:205] Tried to move back new op lower bound from 7023417658735022080 to 7023417658258227200. Current Snapshot: MvccSnapshot[applied={T|T < 7023417658735022080}]
I20240503 01:59:36.883868 10053 tablet.cc:1722] T 574a8dbc94774bd3b2129049d52892bc P 1d0e738c34af40fa9950c6f086619b3a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:36.884917 10053 tablet.cc:1620] T 574a8dbc94774bd3b2129049d52892bc P 1d0e738c34af40fa9950c6f086619b3a: MemRowSet was empty: no flush needed.
I20240503 01:59:36.892936 10054 tablet.cc:1722] T 574a8dbc94774bd3b2129049d52892bc P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:36.894094 6733 catalog_manager.cc:5576] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053) does not have the latest schema for tablet 574a8dbc94774bd3b2129049d52892bc (table rebalancer_dynamic_table [id=03ad9337878a45fe978b426b519046d7]). Expected version 1 got 0
I20240503 01:59:36.894168 10054 tablet.cc:1620] T 574a8dbc94774bd3b2129049d52892bc P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:36.927525 9996 consensus_queue.cc:1035] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:36.937738 8933 consensus_queue.cc:1035] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:36.969826 7013 consensus_queue.cc:237] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:36.969913 6733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:36.971993 6883 raft_consensus.cc:1261] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:36.972386 7275 raft_consensus.cc:1261] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:36.972810 10074 consensus_queue.cc:1035] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:36.973058 9055 consensus_queue.cc:1035] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:36.976678 6948 consensus_peers.cc:485] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef -> Peer db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): Couldn't send request to peer db1a0000aaa346f79c0aa4b6c5105191. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ddc47f7144b4cfdbd9a496b5c7562e8. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:36.976723 9914 tablet.cc:1722] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:36.977227 10054 tablet.cc:1722] T 574a8dbc94774bd3b2129049d52892bc P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:36.977468 9914 tablet.cc:1620] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 01:59:36.976773 9055 raft_consensus.cc:2932] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:36.977748 10054 tablet.cc:1620] T 574a8dbc94774bd3b2129049d52892bc P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:36.979049 7274 raft_consensus.cc:2932] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:36.981216 10053 tablet.cc:1722] T 574a8dbc94774bd3b2129049d52892bc P 1d0e738c34af40fa9950c6f086619b3a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:36.981566 10053 tablet.cc:1620] T 574a8dbc94774bd3b2129049d52892bc P 1d0e738c34af40fa9950c6f086619b3a: MemRowSet was empty: no flush needed.
I20240503 01:59:36.982640 6740 catalog_manager.cc:5484] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:36.984308 6883 raft_consensus.cc:2932] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:36.987833 6733 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=03ad9337878a45fe978b426b519046d7] alter complete (version 2)
I20240503 01:59:37.010813 7275 consensus_queue.cc:237] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:37.012643 6883 raft_consensus.cc:1261] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.012840 7143 raft_consensus.cc:1261] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.013293 9547 consensus_queue.cc:1035] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.013944 9547 consensus_queue.cc:1035] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.016789 8933 raft_consensus.cc:2932] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
W20240503 01:59:37.017522 7211 consensus_peers.cc:485] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9c7396d1b98c4b7ca495ba3ab5dcbbc6. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.018954 6883 raft_consensus.cc:2932] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:37.019980 6740 catalog_manager.cc:5484] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:37.020470 7143 raft_consensus.cc:2932] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:37.044340 7275 consensus_queue.cc:237] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:37.045922 7013 raft_consensus.cc:1261] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.046025 6883 raft_consensus.cc:1261] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:37.046263 7211 consensus_peers.cc:485] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 580e6b4d2b2f4f79b8c2af5e75a3c75f. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.046383 10100 consensus_queue.cc:1035] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.046677 8933 consensus_queue.cc:1035] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.048573 10100 raft_consensus.cc:2932] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:37.049132 7013 raft_consensus.cc:2932] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:37.051563 6882 raft_consensus.cc:2932] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:37.052201 6733 catalog_manager.cc:5484] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:37.079456 7275 consensus_queue.cc:237] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:37.080935 6882 raft_consensus.cc:1261] T e811662836164c3fb29a1b12ace7eb41 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:37.081189 7209 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e811662836164c3fb29a1b12ace7eb41. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.081212 7143 raft_consensus.cc:1261] T e811662836164c3fb29a1b12ace7eb41 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.081454 9996 consensus_queue.cc:1035] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.081631 10100 consensus_queue.cc:1035] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.088382 10100 raft_consensus.cc:2932] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:37.088789 7143 raft_consensus.cc:2932] T e811662836164c3fb29a1b12ace7eb41 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:37.090574 6882 raft_consensus.cc:2932] T e811662836164c3fb29a1b12ace7eb41 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:37.091811 6741 catalog_manager.cc:5484] T e811662836164c3fb29a1b12ace7eb41 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:37.094997 6733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:37.101233 9914 tablet.cc:1722] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:37.104714 10054 tablet.cc:1722] T 574a8dbc94774bd3b2129049d52892bc P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:37.106078 6733 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=03ad9337878a45fe978b426b519046d7] alter complete (version 3)
I20240503 01:59:37.108072 10053 tablet.cc:1722] T 574a8dbc94774bd3b2129049d52892bc P 1d0e738c34af40fa9950c6f086619b3a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:37.132941 7275 consensus_queue.cc:237] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:37.134766 7013 raft_consensus.cc:1261] T c6833c997d284063955d0b9b09ccb3b7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.134809 7143 raft_consensus.cc:1261] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.135205 8933 consensus_queue.cc:1035] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.135396 9996 consensus_queue.cc:1035] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.138022 8933 raft_consensus.cc:2932] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
W20240503 01:59:37.138252 7210 consensus_peers.cc:485] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a -> Peer db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): Couldn't send request to peer db1a0000aaa346f79c0aa4b6c5105191. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6833c997d284063955d0b9b09ccb3b7. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.138495 7143 raft_consensus.cc:2932] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:37.141054 7013 raft_consensus.cc:2932] T c6833c997d284063955d0b9b09ccb3b7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:37.142007 6738 catalog_manager.cc:5484] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:37.165144 7275 consensus_queue.cc:237] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:37.166631 7143 raft_consensus.cc:1261] T a5e23eebc629494cbb8262bc123a720a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:37.166774 7211 consensus_peers.cc:485] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a5e23eebc629494cbb8262bc123a720a. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.166857 7013 raft_consensus.cc:1261] T a5e23eebc629494cbb8262bc123a720a P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.167039 8933 consensus_queue.cc:1035] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.167265 10100 consensus_queue.cc:1035] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.168947 8933 raft_consensus.cc:2932] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:37.169378 7013 raft_consensus.cc:2932] T a5e23eebc629494cbb8262bc123a720a P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:37.171541 7145 raft_consensus.cc:2932] T a5e23eebc629494cbb8262bc123a720a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:37.172226 6733 catalog_manager.cc:5484] T a5e23eebc629494cbb8262bc123a720a P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:37.193468 7275 consensus_queue.cc:237] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:37.194993 7145 raft_consensus.cc:1261] T 836882aa1fac4721be3cf096540eedc4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:37.195231 7209 consensus_peers.cc:485] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 836882aa1fac4721be3cf096540eedc4. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.195262 7013 raft_consensus.cc:1261] T 836882aa1fac4721be3cf096540eedc4 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.195536 10100 consensus_queue.cc:1035] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.195744 8933 consensus_queue.cc:1035] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.197856 10100 raft_consensus.cc:2932] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:37.198249 7013 raft_consensus.cc:2932] T 836882aa1fac4721be3cf096540eedc4 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:37.200014 7145 raft_consensus.cc:2932] T 836882aa1fac4721be3cf096540eedc4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:37.201448 6740 catalog_manager.cc:5484] T 836882aa1fac4721be3cf096540eedc4 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:37.211531 6733 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:37.211769 6733 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:37.214843 6733 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 574a8dbc94774bd3b2129049d52892bc
I20240503 01:59:37.215051 9973 consensus_queue.cc:1035] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:37.215746 9737 tablet_service.cc:1514] Processing DeleteTablet for tablet 574a8dbc94774bd3b2129049d52892bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:37 UTC) from {username='slave'} at 127.0.0.1:48152
I20240503 01:59:37.215827 9342 tablet_service.cc:1514] Processing DeleteTablet for tablet 574a8dbc94774bd3b2129049d52892bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:37 UTC) from {username='slave'} at 127.0.0.1:36040
I20240503 01:59:37.216528 10202 tablet_replica.cc:330] T 574a8dbc94774bd3b2129049d52892bc P db1a0000aaa346f79c0aa4b6c5105191: stopping tablet replica
I20240503 01:59:37.216528 10201 tablet_replica.cc:330] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8: stopping tablet replica
I20240503 01:59:37.216732 10201 raft_consensus.cc:2229] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:37.216754 7254 tablet_service.cc:1514] Processing DeleteTablet for tablet 574a8dbc94774bd3b2129049d52892bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:37 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:37.216965 9952 tablet_replica.cc:330] T 574a8dbc94774bd3b2129049d52892bc P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:37.216995 10201 raft_consensus.cc:2258] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:37.217106 9952 raft_consensus.cc:2229] T 574a8dbc94774bd3b2129049d52892bc P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:37.217228 9952 raft_consensus.cc:2258] T 574a8dbc94774bd3b2129049d52892bc P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:37.217706 10201 ts_tablet_manager.cc:1875] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:37.218326 9952 ts_tablet_manager.cc:1875] T 574a8dbc94774bd3b2129049d52892bc P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:37.219075 10202 raft_consensus.cc:2229] T 574a8dbc94774bd3b2129049d52892bc P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:37.219287 10202 raft_consensus.cc:2258] T 574a8dbc94774bd3b2129049d52892bc P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:37.220407 9952 ts_tablet_manager.cc:1888] T 574a8dbc94774bd3b2129049d52892bc P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:37.220458 9952 log.cc:1196] T 574a8dbc94774bd3b2129049d52892bc P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/574a8dbc94774bd3b2129049d52892bc
I20240503 01:59:37.220795 9952 ts_tablet_manager.cc:1909] T 574a8dbc94774bd3b2129049d52892bc P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:37.220865 10202 ts_tablet_manager.cc:1875] T 574a8dbc94774bd3b2129049d52892bc P db1a0000aaa346f79c0aa4b6c5105191: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:37.221037 10201 ts_tablet_manager.cc:1888] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:37.221156 10201 log.cc:1196] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/574a8dbc94774bd3b2129049d52892bc
I20240503 01:59:37.221475 10201 ts_tablet_manager.cc:1909] T 574a8dbc94774bd3b2129049d52892bc P 12e8c683db054df3b75c6285b98a5ad8: Deleting consensus metadata
I20240503 01:59:37.221688 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet 574a8dbc94774bd3b2129049d52892bc (table rebalancer_dynamic_table_new_name [id=03ad9337878a45fe978b426b519046d7]) successfully deleted
I20240503 01:59:37.222164 6727 catalog_manager.cc:4830] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): tablet 574a8dbc94774bd3b2129049d52892bc (table rebalancer_dynamic_table_new_name [id=03ad9337878a45fe978b426b519046d7]) successfully deleted
I20240503 01:59:37.225160 10202 ts_tablet_manager.cc:1888] T 574a8dbc94774bd3b2129049d52892bc P db1a0000aaa346f79c0aa4b6c5105191: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:37.225270 10202 log.cc:1196] T 574a8dbc94774bd3b2129049d52892bc P db1a0000aaa346f79c0aa4b6c5105191: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/574a8dbc94774bd3b2129049d52892bc
I20240503 01:59:37.225301 9973 consensus_queue.cc:1035] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:37.226377 10202 ts_tablet_manager.cc:1909] T 574a8dbc94774bd3b2129049d52892bc P db1a0000aaa346f79c0aa4b6c5105191: Deleting consensus metadata
I20240503 01:59:37.227774 6726 catalog_manager.cc:4830] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): tablet 574a8dbc94774bd3b2129049d52892bc (table rebalancer_dynamic_table_new_name [id=03ad9337878a45fe978b426b519046d7]) successfully deleted
I20240503 01:59:37.241330 7013 consensus_queue.cc:237] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:37.242995 6882 raft_consensus.cc:1261] T 5cb5072e91534671ad3b4dd85f93fa5b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.243309 7145 raft_consensus.cc:1261] T 5cb5072e91534671ad3b4dd85f93fa5b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.243736 10080 consensus_queue.cc:1035] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.244735 10074 consensus_queue.cc:1035] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:37.246904 6949 consensus_peers.cc:485] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cb5072e91534671ad3b4dd85f93fa5b. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.247012 10074 raft_consensus.cc:2932] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:37.248586 6882 raft_consensus.cc:2932] T 5cb5072e91534671ad3b4dd85f93fa5b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:37.249722 7145 raft_consensus.cc:2932] T 5cb5072e91534671ad3b4dd85f93fa5b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:37.251925 6738 catalog_manager.cc:5484] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:37.273664 6740 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240503 01:59:37.273818 6740 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240503 01:59:37.276233 7013 consensus_queue.cc:237] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:37.276360 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 7e0575a7e09741f68340cb92809831f1
I20240503 01:59:37.276701 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 24462821d97145ba817a4fefbfe1c710
I20240503 01:59:37.277153 7254 tablet_service.cc:1514] Processing DeleteTablet for tablet 24462821d97145ba817a4fefbfe1c710 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:37 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:37.277123 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 6411ac9665194b4f9b9d66f54db0cf25
I20240503 01:59:37.277333 9952 tablet_replica.cc:330] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:37.277503 9952 raft_consensus.cc:2229] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:37.277576 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e0575a7e09741f68340cb92809831f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:37 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:37.277693 7124 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e0575a7e09741f68340cb92809831f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:37 UTC) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:37.277745 9952 raft_consensus.cc:2258] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:37.277774 9955 tablet_replica.cc:330] T 7e0575a7e09741f68340cb92809831f1 P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:37.277958 9955 raft_consensus.cc:2229] T 7e0575a7e09741f68340cb92809831f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:37.278014 10229 tablet_replica.cc:330] T 7e0575a7e09741f68340cb92809831f1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:37.278116 9955 raft_consensus.cc:2258] T 7e0575a7e09741f68340cb92809831f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:37.278295 9952 ts_tablet_manager.cc:1875] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:37.278414 10229 raft_consensus.cc:2229] T 7e0575a7e09741f68340cb92809831f1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:37.278584 10229 raft_consensus.cc:2258] T 7e0575a7e09741f68340cb92809831f1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:37.278592 7254 tablet_service.cc:1514] Processing DeleteTablet for tablet 6411ac9665194b4f9b9d66f54db0cf25 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:37 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:37.279091 10229 ts_tablet_manager.cc:1875] T 7e0575a7e09741f68340cb92809831f1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:37.279176 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet 24462821d97145ba817a4fefbfe1c710 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:37 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:37.279450 7124 tablet_service.cc:1514] Processing DeleteTablet for tablet 6411ac9665194b4f9b9d66f54db0cf25 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:37 UTC) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:37.279780 6992 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e0575a7e09741f68340cb92809831f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:37 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:37.279953 9954 tablet_replica.cc:330] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:37.280088 9954 raft_consensus.cc:2229] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:37.280081 7275 raft_consensus.cc:1261] T c35ecf8890b64caaac7e5ea9f6e8f989 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.280288 9954 raft_consensus.cc:2258] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:37.280782 9954 ts_tablet_manager.cc:1875] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:37.281020 9952 ts_tablet_manager.cc:1888] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:37.281075 9952 log.cc:1196] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/24462821d97145ba817a4fefbfe1c710
I20240503 01:59:37.281283 9952 ts_tablet_manager.cc:1909] T 24462821d97145ba817a4fefbfe1c710 P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:37.281277 10229 ts_tablet_manager.cc:1888] T 7e0575a7e09741f68340cb92809831f1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:37.281340 10229 log.cc:1196] T 7e0575a7e09741f68340cb92809831f1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/7e0575a7e09741f68340cb92809831f1
I20240503 01:59:37.281672 6863 tablet_service.cc:1514] Processing DeleteTablet for tablet 6411ac9665194b4f9b9d66f54db0cf25 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:37 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:37.281936 6882 raft_consensus.cc:1261] T c35ecf8890b64caaac7e5ea9f6e8f989 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.282121 9952 tablet_replica.cc:330] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:37.282261 9952 raft_consensus.cc:2229] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:37.282338 10074 consensus_queue.cc:1035] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.282451 9952 raft_consensus.cc:2258] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:37.282773 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet 24462821d97145ba817a4fefbfe1c710 (table rebalance_test_table_4 [id=636b9e88e8fd430595fb3edfb428c61b]) successfully deleted
I20240503 01:59:37.282968 9952 ts_tablet_manager.cc:1875] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:37.283288 10079 consensus_queue.cc:1035] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.284422 9955 ts_tablet_manager.cc:1875] T 7e0575a7e09741f68340cb92809831f1 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:37.284981 9952 ts_tablet_manager.cc:1888] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:37.285032 9952 log.cc:1196] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/6411ac9665194b4f9b9d66f54db0cf25
I20240503 01:59:37.285231 9952 ts_tablet_manager.cc:1909] T 6411ac9665194b4f9b9d66f54db0cf25 P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:37.285583 10229 ts_tablet_manager.cc:1909] T 7e0575a7e09741f68340cb92809831f1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting consensus metadata
W20240503 01:59:37.285866 6949 consensus_peers.cc:485] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c35ecf8890b64caaac7e5ea9f6e8f989. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.286159 10229 tablet_replica.cc:330] T 6411ac9665194b4f9b9d66f54db0cf25 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:37.286288 10229 raft_consensus.cc:2229] T 6411ac9665194b4f9b9d66f54db0cf25 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:37.286427 10229 raft_consensus.cc:2258] T 6411ac9665194b4f9b9d66f54db0cf25 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:37.286792 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet 6411ac9665194b4f9b9d66f54db0cf25 (table rebalance_test_table_4 [id=636b9e88e8fd430595fb3edfb428c61b]) successfully deleted
I20240503 01:59:37.286935 10229 ts_tablet_manager.cc:1875] T 6411ac9665194b4f9b9d66f54db0cf25 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:37.288220 6740 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:37.288782 9954 ts_tablet_manager.cc:1888] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:37.288863 9954 log.cc:1196] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/7e0575a7e09741f68340cb92809831f1
I20240503 01:59:37.289103 9954 ts_tablet_manager.cc:1909] T 7e0575a7e09741f68340cb92809831f1 P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:37.291043 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet 7e0575a7e09741f68340cb92809831f1 (table rebalance_test_table_4 [id=636b9e88e8fd430595fb3edfb428c61b]) successfully deleted
I20240503 01:59:37.291211 10229 ts_tablet_manager.cc:1888] T 6411ac9665194b4f9b9d66f54db0cf25 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:37.291278 10229 log.cc:1196] T 6411ac9665194b4f9b9d66f54db0cf25 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/6411ac9665194b4f9b9d66f54db0cf25
I20240503 01:59:37.291543 10229 ts_tablet_manager.cc:1909] T 6411ac9665194b4f9b9d66f54db0cf25 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting consensus metadata
I20240503 01:59:37.292217 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet 6411ac9665194b4f9b9d66f54db0cf25 (table rebalance_test_table_4 [id=636b9e88e8fd430595fb3edfb428c61b]) successfully deleted
I20240503 01:59:37.292824 6992 tablet_service.cc:1514] Processing DeleteTablet for tablet 24462821d97145ba817a4fefbfe1c710 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:37 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:37.292896 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet 7e0575a7e09741f68340cb92809831f1 (table rebalance_test_table_4 [id=636b9e88e8fd430595fb3edfb428c61b]) successfully deleted
I20240503 01:59:37.293000 9954 tablet_replica.cc:330] T 24462821d97145ba817a4fefbfe1c710 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:37.292959 10074 raft_consensus.cc:2932] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:37.293139 9954 raft_consensus.cc:2229] T 24462821d97145ba817a4fefbfe1c710 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:37.293283 9954 raft_consensus.cc:2258] T 24462821d97145ba817a4fefbfe1c710 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:37.293803 9954 ts_tablet_manager.cc:1875] T 24462821d97145ba817a4fefbfe1c710 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:37.294566 7275 raft_consensus.cc:2932] T c35ecf8890b64caaac7e5ea9f6e8f989 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:37.295828 9954 ts_tablet_manager.cc:1888] T 24462821d97145ba817a4fefbfe1c710 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:37.295889 9954 log.cc:1196] T 24462821d97145ba817a4fefbfe1c710 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/24462821d97145ba817a4fefbfe1c710
I20240503 01:59:37.295817 9737 tablet_service.cc:1467] Processing CreateTablet for tablet 1d182ce2c5084082a0990ce8090eb88b (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=cbc39ac1f52f44debbf6d5a5d0edd3ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:37.296098 9737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d182ce2c5084082a0990ce8090eb88b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.297075 9736 tablet_service.cc:1467] Processing CreateTablet for tablet fb287629baab4cada193ae57695ef1b5 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=cbc39ac1f52f44debbf6d5a5d0edd3ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:37.297322 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb287629baab4cada193ae57695ef1b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.298398 10232 tablet_bootstrap.cc:492] T 1d182ce2c5084082a0990ce8090eb88b P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:37.299569 6992 tablet_service.cc:1467] Processing CreateTablet for tablet fb287629baab4cada193ae57695ef1b5 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=cbc39ac1f52f44debbf6d5a5d0edd3ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:37.299649 10232 tablet_bootstrap.cc:654] T 1d182ce2c5084082a0990ce8090eb88b P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:37.299845 6992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb287629baab4cada193ae57695ef1b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.300120 9954 ts_tablet_manager.cc:1909] T 24462821d97145ba817a4fefbfe1c710 P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:37.300823 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet 24462821d97145ba817a4fefbfe1c710 (table rebalance_test_table_4 [id=636b9e88e8fd430595fb3edfb428c61b]) successfully deleted
I20240503 01:59:37.300922 6733 catalog_manager.cc:5484] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:37.301625 10234 tablet_bootstrap.cc:492] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:37.301651 9606 tablet_service.cc:1467] Processing CreateTablet for tablet 1d182ce2c5084082a0990ce8090eb88b (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=cbc39ac1f52f44debbf6d5a5d0edd3ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:37.301908 9606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d182ce2c5084082a0990ce8090eb88b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.302482 10234 tablet_bootstrap.cc:654] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:37.303516 10234 tablet_bootstrap.cc:492] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:37.303612 10234 ts_tablet_manager.cc:1367] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:37.303964 10236 tablet_bootstrap.cc:492] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:37.304118 9605 tablet_service.cc:1467] Processing CreateTablet for tablet fb287629baab4cada193ae57695ef1b5 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=cbc39ac1f52f44debbf6d5a5d0edd3ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:37.304188 10234 raft_consensus.cc:348] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:37.304340 10234 raft_consensus.cc:374] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:37.304358 9605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb287629baab4cada193ae57695ef1b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.304412 10234 raft_consensus.cc:729] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:37.304585 10234 consensus_queue.cc:260] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:37.305038 10234 ts_tablet_manager.cc:1398] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:37.305291 10236 tablet_bootstrap.cc:654] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:37.306406 9604 tablet_service.cc:1467] Processing CreateTablet for tablet 54e3860e5cf54663b4538b9e0f96664e (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=cbc39ac1f52f44debbf6d5a5d0edd3ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:37.306689 10236 tablet_bootstrap.cc:492] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:37.306687 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e3860e5cf54663b4538b9e0f96664e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.306844 10236 ts_tablet_manager.cc:1367] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:37.306973 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 1d182ce2c5084082a0990ce8090eb88b (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=cbc39ac1f52f44debbf6d5a5d0edd3ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:37.307253 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d182ce2c5084082a0990ce8090eb88b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.307360 10236 raft_consensus.cc:348] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:37.307457 10236 raft_consensus.cc:374] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:37.307487 10236 raft_consensus.cc:729] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:37.307593 10236 consensus_queue.cc:260] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:37.308310 10236 ts_tablet_manager.cc:1398] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:37.308498 10236 tablet_bootstrap.cc:492] T fb287629baab4cada193ae57695ef1b5 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:37.309327 10238 tablet_bootstrap.cc:492] T 1d182ce2c5084082a0990ce8090eb88b P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:37.309522 10236 tablet_bootstrap.cc:654] T fb287629baab4cada193ae57695ef1b5 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:37.309535 9343 tablet_service.cc:1467] Processing CreateTablet for tablet 54e3860e5cf54663b4538b9e0f96664e (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=cbc39ac1f52f44debbf6d5a5d0edd3ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:37.309734 9343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e3860e5cf54663b4538b9e0f96664e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.310215 10238 tablet_bootstrap.cc:654] T 1d182ce2c5084082a0990ce8090eb88b P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:37.311690 10238 tablet_bootstrap.cc:492] T 1d182ce2c5084082a0990ce8090eb88b P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:37.311776 10238 ts_tablet_manager.cc:1367] T 1d182ce2c5084082a0990ce8090eb88b P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:37.312206 10238 raft_consensus.cc:348] T 1d182ce2c5084082a0990ce8090eb88b P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:37.312345 10238 raft_consensus.cc:374] T 1d182ce2c5084082a0990ce8090eb88b P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:37.312403 10238 raft_consensus.cc:729] T 1d182ce2c5084082a0990ce8090eb88b P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:37.312520 10238 consensus_queue.cc:260] T 1d182ce2c5084082a0990ce8090eb88b P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:37.286510 9955 ts_tablet_manager.cc:1888] T 7e0575a7e09741f68340cb92809831f1 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:37.312983 9955 log.cc:1196] T 7e0575a7e09741f68340cb92809831f1 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/7e0575a7e09741f68340cb92809831f1
I20240503 01:59:37.298888 6863 tablet_service.cc:1467] Processing CreateTablet for tablet 54e3860e5cf54663b4538b9e0f96664e (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=cbc39ac1f52f44debbf6d5a5d0edd3ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:37.313870 9955 ts_tablet_manager.cc:1909] T 7e0575a7e09741f68340cb92809831f1 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:37.313997 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e3860e5cf54663b4538b9e0f96664e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.314163 10238 ts_tablet_manager.cc:1398] T 1d182ce2c5084082a0990ce8090eb88b P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:37.314313 10238 tablet_bootstrap.cc:492] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:37.314577 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 7e0575a7e09741f68340cb92809831f1 (table rebalance_test_table_4 [id=636b9e88e8fd430595fb3edfb428c61b]) successfully deleted
I20240503 01:59:37.315843 10238 tablet_bootstrap.cc:654] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:37.299109 6882 raft_consensus.cc:2932] T c35ecf8890b64caaac7e5ea9f6e8f989 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:37.317129 9955 tablet_replica.cc:330] T 24462821d97145ba817a4fefbfe1c710 P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:37.317140 6740 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:37.317291 9955 raft_consensus.cc:2229] T 24462821d97145ba817a4fefbfe1c710 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:37.317466 9955 raft_consensus.cc:2258] T 24462821d97145ba817a4fefbfe1c710 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:37.317968 9955 ts_tablet_manager.cc:1875] T 24462821d97145ba817a4fefbfe1c710 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:37.320788 10241 tablet_bootstrap.cc:492] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:37.321784 10241 tablet_bootstrap.cc:654] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:37.324455 9343 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa702eb78ae401f825c945567e8ac3e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7d48947bbe7d49fc92bed1b29c44a9fe]), partition=
I20240503 01:59:37.324569 9736 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa702eb78ae401f825c945567e8ac3e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7d48947bbe7d49fc92bed1b29c44a9fe]), partition=
I20240503 01:59:37.324741 9343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa702eb78ae401f825c945567e8ac3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.324865 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa702eb78ae401f825c945567e8ac3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.326891 9474 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa702eb78ae401f825c945567e8ac3e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7d48947bbe7d49fc92bed1b29c44a9fe]), partition=
I20240503 01:59:37.327128 9474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa702eb78ae401f825c945567e8ac3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.324203 9955 ts_tablet_manager.cc:1888] T 24462821d97145ba817a4fefbfe1c710 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:37.330744 9955 log.cc:1196] T 24462821d97145ba817a4fefbfe1c710 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/24462821d97145ba817a4fefbfe1c710
I20240503 01:59:37.331099 9955 ts_tablet_manager.cc:1909] T 24462821d97145ba817a4fefbfe1c710 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:37.331085 10244 tablet_bootstrap.cc:492] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:37.333675 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 24462821d97145ba817a4fefbfe1c710 (table rebalance_test_table_4 [id=636b9e88e8fd430595fb3edfb428c61b]) successfully deleted
I20240503 01:59:37.333966 10244 tablet_bootstrap.cc:654] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:37.335045 10244 tablet_bootstrap.cc:492] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:37.335227 10244 ts_tablet_manager.cc:1367] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:37.335453 9955 tablet_replica.cc:330] T 6411ac9665194b4f9b9d66f54db0cf25 P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:37.335662 9955 raft_consensus.cc:2229] T 6411ac9665194b4f9b9d66f54db0cf25 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:37.335794 9955 raft_consensus.cc:2258] T 6411ac9665194b4f9b9d66f54db0cf25 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:37.336323 9955 ts_tablet_manager.cc:1875] T 6411ac9665194b4f9b9d66f54db0cf25 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:37.325148 10241 tablet_bootstrap.cc:492] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:37.336529 10241 ts_tablet_manager.cc:1367] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 01:59:37.336723 10244 raft_consensus.cc:348] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:37.336833 10244 raft_consensus.cc:374] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:37.336891 10244 raft_consensus.cc:729] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:37.336977 10241 raft_consensus.cc:348] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:37.337088 10241 raft_consensus.cc:374] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:37.337071 10244 consensus_queue.cc:260] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:37.337384 10244 ts_tablet_manager.cc:1398] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:37.337683 10238 tablet_bootstrap.cc:492] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:37.337785 10238 ts_tablet_manager.cc:1367] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.000s
I20240503 01:59:37.338284 10238 raft_consensus.cc:348] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:37.338393 10238 raft_consensus.cc:374] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:37.338424 10238 raft_consensus.cc:729] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:37.338585 10238 consensus_queue.cc:260] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:37.339740 10232 tablet_bootstrap.cc:492] T 1d182ce2c5084082a0990ce8090eb88b P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:37.339767 10236 tablet_bootstrap.cc:492] T fb287629baab4cada193ae57695ef1b5 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:37.339851 10232 ts_tablet_manager.cc:1367] T 1d182ce2c5084082a0990ce8090eb88b P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.000s
I20240503 01:59:37.339851 10236 ts_tablet_manager.cc:1367] T fb287629baab4cada193ae57695ef1b5 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.000s
I20240503 01:59:37.340265 10238 ts_tablet_manager.cc:1398] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:37.340327 10241 raft_consensus.cc:729] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:37.340308 10236 raft_consensus.cc:348] T fb287629baab4cada193ae57695ef1b5 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:37.340293 10232 raft_consensus.cc:348] T 1d182ce2c5084082a0990ce8090eb88b P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:37.340399 10232 raft_consensus.cc:374] T 1d182ce2c5084082a0990ce8090eb88b P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:37.340399 10236 raft_consensus.cc:374] T fb287629baab4cada193ae57695ef1b5 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:37.340431 10232 raft_consensus.cc:729] T 1d182ce2c5084082a0990ce8090eb88b P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:37.340431 10236 raft_consensus.cc:729] T fb287629baab4cada193ae57695ef1b5 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:37.340472 10241 consensus_queue.cc:260] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:37.340551 10232 consensus_queue.cc:260] T 1d182ce2c5084082a0990ce8090eb88b P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:37.340550 10236 consensus_queue.cc:260] T fb287629baab4cada193ae57695ef1b5 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:37.340801 10232 ts_tablet_manager.cc:1398] T 1d182ce2c5084082a0990ce8090eb88b P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:37.340802 10241 ts_tablet_manager.cc:1398] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:37.340812 10236 ts_tablet_manager.cc:1398] T fb287629baab4cada193ae57695ef1b5 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:37.340973 10232 tablet_bootstrap.cc:492] T fb287629baab4cada193ae57695ef1b5 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:37.340974 10236 tablet_bootstrap.cc:492] T 54e3860e5cf54663b4538b9e0f96664e P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:37.341889 10232 tablet_bootstrap.cc:654] T fb287629baab4cada193ae57695ef1b5 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:37.341889 10236 tablet_bootstrap.cc:654] T 54e3860e5cf54663b4538b9e0f96664e P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:37.342741 9955 ts_tablet_manager.cc:1888] T 6411ac9665194b4f9b9d66f54db0cf25 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:37.342810 9955 log.cc:1196] T 6411ac9665194b4f9b9d66f54db0cf25 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/6411ac9665194b4f9b9d66f54db0cf25
I20240503 01:59:37.343094 9955 ts_tablet_manager.cc:1909] T 6411ac9665194b4f9b9d66f54db0cf25 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:37.343192 10238 tablet_bootstrap.cc:492] T 9aa702eb78ae401f825c945567e8ac3e P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:37.345085 10238 tablet_bootstrap.cc:654] T 9aa702eb78ae401f825c945567e8ac3e P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:37.345309 10236 tablet_bootstrap.cc:492] T 54e3860e5cf54663b4538b9e0f96664e P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:37.345399 10236 ts_tablet_manager.cc:1367] T 54e3860e5cf54663b4538b9e0f96664e P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:37.345939 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 6411ac9665194b4f9b9d66f54db0cf25 (table rebalance_test_table_4 [id=636b9e88e8fd430595fb3edfb428c61b]) successfully deleted
I20240503 01:59:37.345921 10236 raft_consensus.cc:348] T 54e3860e5cf54663b4538b9e0f96664e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:37.346036 10236 raft_consensus.cc:374] T 54e3860e5cf54663b4538b9e0f96664e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:37.346079 10236 raft_consensus.cc:729] T 54e3860e5cf54663b4538b9e0f96664e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:37.346205 10236 consensus_queue.cc:260] T 54e3860e5cf54663b4538b9e0f96664e P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:37.346511 10236 ts_tablet_manager.cc:1398] T 54e3860e5cf54663b4538b9e0f96664e P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:37.348656 10232 tablet_bootstrap.cc:492] T fb287629baab4cada193ae57695ef1b5 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:37.348758 10232 ts_tablet_manager.cc:1367] T fb287629baab4cada193ae57695ef1b5 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 01:59:37.349310 10232 raft_consensus.cc:348] T fb287629baab4cada193ae57695ef1b5 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:37.349421 10232 raft_consensus.cc:374] T fb287629baab4cada193ae57695ef1b5 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:37.349455 10232 raft_consensus.cc:729] T fb287629baab4cada193ae57695ef1b5 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:37.349589 10232 consensus_queue.cc:260] T fb287629baab4cada193ae57695ef1b5 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:37.349896 10232 ts_tablet_manager.cc:1398] T fb287629baab4cada193ae57695ef1b5 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:37.350096 10232 tablet_bootstrap.cc:492] T 9aa702eb78ae401f825c945567e8ac3e P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:37.352082 10232 tablet_bootstrap.cc:654] T 9aa702eb78ae401f825c945567e8ac3e P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:37.353502 10232 tablet_bootstrap.cc:492] T 9aa702eb78ae401f825c945567e8ac3e P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:37.353600 10232 ts_tablet_manager.cc:1367] T 9aa702eb78ae401f825c945567e8ac3e P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 01:59:37.353979 10232 raft_consensus.cc:348] T 9aa702eb78ae401f825c945567e8ac3e P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:37.354137 10232 raft_consensus.cc:374] T 9aa702eb78ae401f825c945567e8ac3e P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:37.354173 10232 raft_consensus.cc:729] T 9aa702eb78ae401f825c945567e8ac3e P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:37.354321 10232 consensus_queue.cc:260] T 9aa702eb78ae401f825c945567e8ac3e P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:37.354604 10232 ts_tablet_manager.cc:1398] T 9aa702eb78ae401f825c945567e8ac3e P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:37.356387 10238 tablet_bootstrap.cc:492] T 9aa702eb78ae401f825c945567e8ac3e P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:37.356493 10238 ts_tablet_manager.cc:1367] T 9aa702eb78ae401f825c945567e8ac3e P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 01:59:37.357095 10238 raft_consensus.cc:348] T 9aa702eb78ae401f825c945567e8ac3e P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:37.357214 10238 raft_consensus.cc:374] T 9aa702eb78ae401f825c945567e8ac3e P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:37.357242 10238 raft_consensus.cc:729] T 9aa702eb78ae401f825c945567e8ac3e P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:37.357363 10238 consensus_queue.cc:260] T 9aa702eb78ae401f825c945567e8ac3e P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:37.358394 10238 ts_tablet_manager.cc:1398] T 9aa702eb78ae401f825c945567e8ac3e P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:37.378924 6882 consensus_queue.cc:237] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:37.381510 7145 raft_consensus.cc:1261] T dec7e3036d384f5db51f50ba93acbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.381932 7275 raft_consensus.cc:1261] T dec7e3036d384f5db51f50ba93acbbc6 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.382131 10267 consensus_queue.cc:1035] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.383128 10267 consensus_queue.cc:1035] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.383765 9975 raft_consensus.cc:482] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:37.384056 9975 raft_consensus.cc:504] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:37.385368 9975 leader_election.cc:288] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
W20240503 01:59:37.385816 6817 consensus_peers.cc:485] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): Couldn't send request to peer db1a0000aaa346f79c0aa4b6c5105191. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dec7e3036d384f5db51f50ba93acbbc6. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.388482 10029 raft_consensus.cc:2932] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:37.388864 7275 raft_consensus.cc:2932] T dec7e3036d384f5db51f50ba93acbbc6 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:37.390161 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa702eb78ae401f825c945567e8ac3e" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:37.390300 9363 raft_consensus.cc:2454] T 9aa702eb78ae401f825c945567e8ac3e P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:37.390703 9429 leader_election.cc:302] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:37.390967 9975 raft_consensus.cc:2781] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:37.391019 9975 raft_consensus.cc:482] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:37.391054 9975 raft_consensus.cc:3037] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:37.391541 7145 raft_consensus.cc:2932] T dec7e3036d384f5db51f50ba93acbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:37.392107 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa702eb78ae401f825c945567e8ac3e" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:37.392242 9757 raft_consensus.cc:2454] T 9aa702eb78ae401f825c945567e8ac3e P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:37.392242 9975 raft_consensus.cc:504] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:37.392623 9975 leader_election.cc:288] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Requested vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:37.392997 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa702eb78ae401f825c945567e8ac3e" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:37.393108 9757 raft_consensus.cc:3037] T 9aa702eb78ae401f825c945567e8ac3e P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:37.393311 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa702eb78ae401f825c945567e8ac3e" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:37.393229 6738 catalog_manager.cc:5484] T dec7e3036d384f5db51f50ba93acbbc6 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:37.393414 9363 raft_consensus.cc:3037] T 9aa702eb78ae401f825c945567e8ac3e P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:37.394109 9757 raft_consensus.cc:2454] T 9aa702eb78ae401f825c945567e8ac3e P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:37.394424 9363 raft_consensus.cc:2454] T 9aa702eb78ae401f825c945567e8ac3e P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:37.394784 9428 leader_election.cc:302] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:37.397171 9975 raft_consensus.cc:2781] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:37.397462 9975 raft_consensus.cc:686] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Becoming Leader. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Running, Role: LEADER
I20240503 01:59:37.397815 9975 consensus_queue.cc:237] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:37.400130 6738 catalog_manager.cc:5484] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: term changed from 0 to 1, leader changed from <none> to bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:37.421450 10292 ts_tablet_manager.cc:902] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 12e8c683db054df3b75c6285b98a5ad8: Initiating tablet copy from peer 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:37.422302 10292 tablet_copy_client.cc:308] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36271
I20240503 01:59:37.422199 7145 consensus_queue.cc:237] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:37.424341 7296 tablet_copy_service.cc:140] P 1d0e738c34af40fa9950c6f086619b3a: Received BeginTabletCopySession request for tablet 9c7396d1b98c4b7ca495ba3ab5dcbbc6 from peer 12e8c683db054df3b75c6285b98a5ad8 ({username='slave'} at 127.0.99.133:37177)
I20240503 01:59:37.424500 7296 tablet_copy_service.cc:161] P 1d0e738c34af40fa9950c6f086619b3a: Beginning new tablet copy session on tablet 9c7396d1b98c4b7ca495ba3ab5dcbbc6 from peer 12e8c683db054df3b75c6285b98a5ad8 at {username='slave'} at 127.0.99.133:37177: session id = 12e8c683db054df3b75c6285b98a5ad8-9c7396d1b98c4b7ca495ba3ab5dcbbc6
I20240503 01:59:37.424674 7013 raft_consensus.cc:1261] T ccbd932e20b44dc5aceb86c86d7fda6b P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.425238 9092 consensus_queue.cc:1035] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.425956 7296 tablet_copy_source_session.cc:205] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.426966 7275 raft_consensus.cc:1261] T ccbd932e20b44dc5aceb86c86d7fda6b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.427944 10292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7396d1b98c4b7ca495ba3ab5dcbbc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.430186 10292 tablet_copy_client.cc:783] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.430368 10292 tablet_copy_client.cc:647] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.430661 10298 consensus_queue.cc:1035] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:37.432003 7080 consensus_peers.cc:485] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ccbd932e20b44dc5aceb86c86d7fda6b. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.433887 10292 tablet_copy_client.cc:523] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.433830 9092 raft_consensus.cc:2932] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:37.435170 7013 raft_consensus.cc:2932] T ccbd932e20b44dc5aceb86c86d7fda6b P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:37.435593 10292 tablet_bootstrap.cc:492] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:37.436676 10078 raft_consensus.cc:482] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:37.436756 10078 raft_consensus.cc:504] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:37.437171 10078 leader_election.cc:288] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:37.439129 6738 catalog_manager.cc:5484] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:37.442766 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e3860e5cf54663b4538b9e0f96664e" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:37.442900 9625 raft_consensus.cc:2454] T 54e3860e5cf54663b4538b9e0f96664e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:37.443161 9298 leader_election.cc:302] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:37.443341 10078 raft_consensus.cc:2781] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:37.443389 10078 raft_consensus.cc:482] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:37.443425 10078 raft_consensus.cc:3037] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:37.443756 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e3860e5cf54663b4538b9e0f96664e" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:37.443879 6882 raft_consensus.cc:2454] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:37.444380 10078 raft_consensus.cc:504] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:37.444737 10078 leader_election.cc:288] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Requested vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:37.445077 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e3860e5cf54663b4538b9e0f96664e" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:37.445183 6882 raft_consensus.cc:3037] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:37.445155 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e3860e5cf54663b4538b9e0f96664e" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:37.445281 9625 raft_consensus.cc:3037] T 54e3860e5cf54663b4538b9e0f96664e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:37.446158 9625 raft_consensus.cc:2454] T 54e3860e5cf54663b4538b9e0f96664e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:37.446177 6882 raft_consensus.cc:2454] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:37.446462 9299 leader_election.cc:302] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:37.446674 10078 raft_consensus.cc:2781] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:37.446769 10078 raft_consensus.cc:686] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Running, Role: LEADER
I20240503 01:59:37.446921 10078 consensus_queue.cc:237] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:37.449095 6733 catalog_manager.cc:5484] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:37.449877 7275 raft_consensus.cc:2932] T ccbd932e20b44dc5aceb86c86d7fda6b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:37.467746 10292 tablet_bootstrap.cc:492] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:37.468055 10292 tablet_bootstrap.cc:492] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap complete.
I20240503 01:59:37.468261 10292 ts_tablet_manager.cc:1367] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.033s user 0.023s sys 0.007s
I20240503 01:59:37.468679 10292 raft_consensus.cc:348] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:37.468788 10292 raft_consensus.cc:729] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:37.468923 10292 consensus_queue.cc:260] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:37.469213 10292 ts_tablet_manager.cc:1398] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20240503 01:59:37.469846 7296 tablet_copy_service.cc:342] P 1d0e738c34af40fa9950c6f086619b3a: Request end of tablet copy session 12e8c683db054df3b75c6285b98a5ad8-9c7396d1b98c4b7ca495ba3ab5dcbbc6 received from {username='slave'} at 127.0.99.133:37177
I20240503 01:59:37.469946 7296 tablet_copy_service.cc:433] P 1d0e738c34af40fa9950c6f086619b3a: ending tablet copy session 12e8c683db054df3b75c6285b98a5ad8-9c7396d1b98c4b7ca495ba3ab5dcbbc6 on tablet 9c7396d1b98c4b7ca495ba3ab5dcbbc6 with peer 12e8c683db054df3b75c6285b98a5ad8
I20240503 01:59:37.471132 7145 consensus_queue.cc:237] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:37.472786 6882 raft_consensus.cc:1261] T 8ed2793842464443826f83f60fb6dcd1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.473145 9824 consensus_queue.cc:1035] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.473954 7013 raft_consensus.cc:1261] T 8ed2793842464443826f83f60fb6dcd1 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.474474 9824 consensus_queue.cc:1035] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.477036 10302 raft_consensus.cc:2932] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:37.477365 7013 raft_consensus.cc:2932] T 8ed2793842464443826f83f60fb6dcd1 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:37.481272 6733 catalog_manager.cc:5484] T 8ed2793842464443826f83f60fb6dcd1 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:37.481846 6882 raft_consensus.cc:2932] T 8ed2793842464443826f83f60fb6dcd1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
W20240503 01:59:37.487792 7078 consensus_peers.cc:485] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ed2793842464443826f83f60fb6dcd1. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.507484 7013 consensus_queue.cc:237] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:37.509202 7275 raft_consensus.cc:1261] T 49b580d648014ce899fee8b3b329c47d P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.509454 6882 raft_consensus.cc:1261] T 49b580d648014ce899fee8b3b329c47d P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.509845 9055 consensus_queue.cc:1035] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.510018 10079 consensus_queue.cc:1035] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:37.510221 6949 consensus_peers.cc:485] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49b580d648014ce899fee8b3b329c47d. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.512282 6738 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:37.518815 9363 raft_consensus.cc:1261] T 9aa702eb78ae401f825c945567e8ac3e P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:37.518867 9757 raft_consensus.cc:1261] T 9aa702eb78ae401f825c945567e8ac3e P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:37.519364 9975 consensus_queue.cc:1035] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:37.519627 10274 consensus_queue.cc:1035] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:37.520190 10342 ts_tablet_manager.cc:902] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P b7062fbbc2b54fff9e99ec4ea5562c76: Initiating tablet copy from peer 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:37.521111 10342 tablet_copy_client.cc:308] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36271
I20240503 01:59:37.522905 6738 catalog_manager.cc:5576] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053) does not have the latest schema for tablet 9aa702eb78ae401f825c945567e8ac3e (table rebalancer_dynamic_table [id=7d48947bbe7d49fc92bed1b29c44a9fe]). Expected version 1 got 0
I20240503 01:59:37.523579 6740 catalog_manager.cc:5576] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387) does not have the latest schema for tablet 9aa702eb78ae401f825c945567e8ac3e (table rebalancer_dynamic_table [id=7d48947bbe7d49fc92bed1b29c44a9fe]). Expected version 1 got 0
I20240503 01:59:37.524217 10350 ts_tablet_manager.cc:902] T e811662836164c3fb29a1b12ace7eb41 P bb5364d730cb4489a0bfc6c270b7450a: Initiating tablet copy from peer 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:37.524588 9055 raft_consensus.cc:2932] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:37.525070 10350 tablet_copy_client.cc:308] T e811662836164c3fb29a1b12ace7eb41 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36271
I20240503 01:59:37.525081 6882 raft_consensus.cc:2932] T 49b580d648014ce899fee8b3b329c47d P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:37.528141 6740 catalog_manager.cc:5484] T 49b580d648014ce899fee8b3b329c47d P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:37.529099 7296 tablet_copy_service.cc:140] P 1d0e738c34af40fa9950c6f086619b3a: Received BeginTabletCopySession request for tablet 580e6b4d2b2f4f79b8c2af5e75a3c75f from peer b7062fbbc2b54fff9e99ec4ea5562c76 ({username='slave'} at 127.0.99.135:51921)
I20240503 01:59:37.529225 7296 tablet_copy_service.cc:161] P 1d0e738c34af40fa9950c6f086619b3a: Beginning new tablet copy session on tablet 580e6b4d2b2f4f79b8c2af5e75a3c75f from peer b7062fbbc2b54fff9e99ec4ea5562c76 at {username='slave'} at 127.0.99.135:51921: session id = b7062fbbc2b54fff9e99ec4ea5562c76-580e6b4d2b2f4f79b8c2af5e75a3c75f
I20240503 01:59:37.529433 7275 raft_consensus.cc:2932] T 49b580d648014ce899fee8b3b329c47d P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:37.530208 7296 tablet_copy_source_session.cc:205] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.532429 10342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 580e6b4d2b2f4f79b8c2af5e75a3c75f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.536226 9914 tablet.cc:1722] T 9aa702eb78ae401f825c945567e8ac3e P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:37.536569 9914 tablet.cc:1620] T 9aa702eb78ae401f825c945567e8ac3e P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 01:59:37.537350 10342 tablet_copy_client.cc:783] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.537629 10342 tablet_copy_client.cc:647] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.538559 10046 mvcc.cc:205] Tried to move back new op lower bound from 7023417661508669440 to 7023417661024768000. Current Snapshot: MvccSnapshot[applied={T|T < 7023417661508669440}]
I20240503 01:59:37.538777 10054 tablet.cc:1722] T 9aa702eb78ae401f825c945567e8ac3e P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:37.539077 10054 tablet.cc:1620] T 9aa702eb78ae401f825c945567e8ac3e P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:37.539891 6740 catalog_manager.cc:5576] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053) does not have the latest schema for tablet 9aa702eb78ae401f825c945567e8ac3e (table rebalancer_dynamic_table [id=7d48947bbe7d49fc92bed1b29c44a9fe]). Expected version 1 got 0
I20240503 01:59:37.540292 10079 raft_consensus.cc:482] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:37.540388 10079 raft_consensus.cc:504] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:37.540463 6741 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=7d48947bbe7d49fc92bed1b29c44a9fe] alter complete (version 1)
I20240503 01:59:37.540784 10079 leader_election.cc:288] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:37.541188 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb287629baab4cada193ae57695ef1b5" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:37.541327 9625 raft_consensus.cc:2454] T fb287629baab4cada193ae57695ef1b5 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:37.541754 6949 leader_election.cc:302] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:37.542157 10079 raft_consensus.cc:2781] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:37.542212 10079 raft_consensus.cc:482] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:37.542248 10079 raft_consensus.cc:3037] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:37.542289 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb287629baab4cada193ae57695ef1b5" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:37.542402 9757 raft_consensus.cc:2454] T fb287629baab4cada193ae57695ef1b5 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:37.543334 10079 raft_consensus.cc:504] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:37.544528 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb287629baab4cada193ae57695ef1b5" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:37.544646 9625 raft_consensus.cc:3037] T fb287629baab4cada193ae57695ef1b5 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:37.545549 9625 raft_consensus.cc:2454] T fb287629baab4cada193ae57695ef1b5 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:37.545718 10079 leader_election.cc:288] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:37.545796 6949 leader_election.cc:302] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:37.546008 10079 raft_consensus.cc:2781] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:37.546211 10079 raft_consensus.cc:686] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:37.546311 7296 tablet_copy_service.cc:140] P 1d0e738c34af40fa9950c6f086619b3a: Received BeginTabletCopySession request for tablet e811662836164c3fb29a1b12ace7eb41 from peer bb5364d730cb4489a0bfc6c270b7450a ({username='slave'} at 127.0.99.134:48821)
I20240503 01:59:37.546402 7296 tablet_copy_service.cc:161] P 1d0e738c34af40fa9950c6f086619b3a: Beginning new tablet copy session on tablet e811662836164c3fb29a1b12ace7eb41 from peer bb5364d730cb4489a0bfc6c270b7450a at {username='slave'} at 127.0.99.134:48821: session id = bb5364d730cb4489a0bfc6c270b7450a-e811662836164c3fb29a1b12ace7eb41
I20240503 01:59:37.546365 10079 consensus_queue.cc:237] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:37.547379 7296 tablet_copy_source_session.cc:205] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.548856 6741 catalog_manager.cc:5484] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:37.549211 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb287629baab4cada193ae57695ef1b5" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:37.549324 9757 raft_consensus.cc:3037] T fb287629baab4cada193ae57695ef1b5 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:37.550009 9757 raft_consensus.cc:2454] T fb287629baab4cada193ae57695ef1b5 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:37.551805 10350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e811662836164c3fb29a1b12ace7eb41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.551874 10360 tablet.cc:1722] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:37.552628 10360 tablet.cc:1620] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:37.553973 10350 tablet_copy_client.cc:783] T e811662836164c3fb29a1b12ace7eb41 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.554172 10350 tablet_copy_client.cc:647] T e811662836164c3fb29a1b12ace7eb41 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.555846 10342 tablet_copy_client.cc:523] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.556020 10350 tablet_copy_client.cc:523] T e811662836164c3fb29a1b12ace7eb41 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.557612 10342 tablet_bootstrap.cc:492] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:37.557626 10350 tablet_bootstrap.cc:492] T e811662836164c3fb29a1b12ace7eb41 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:37.562732 10365 ts_tablet_manager.cc:902] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P db1a0000aaa346f79c0aa4b6c5105191: Initiating tablet copy from peer 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:37.563458 10365 tablet_copy_client.cc:308] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:37231
I20240503 01:59:37.565752 9970 raft_consensus.cc:482] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:37.565878 9970 raft_consensus.cc:504] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:37.567047 9970 leader_election.cc:288] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:37.570510 7034 tablet_copy_service.cc:140] P 63331f455be049c1bdd6c23c59ac1bef: Received BeginTabletCopySession request for tablet 5ddc47f7144b4cfdbd9a496b5c7562e8 from peer db1a0000aaa346f79c0aa4b6c5105191 ({username='slave'} at 127.0.99.136:38763)
I20240503 01:59:37.570683 7034 tablet_copy_service.cc:161] P 63331f455be049c1bdd6c23c59ac1bef: Beginning new tablet copy session on tablet 5ddc47f7144b4cfdbd9a496b5c7562e8 from peer db1a0000aaa346f79c0aa4b6c5105191 at {username='slave'} at 127.0.99.136:38763: session id = db1a0000aaa346f79c0aa4b6c5105191-5ddc47f7144b4cfdbd9a496b5c7562e8
I20240503 01:59:37.572383 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d182ce2c5084082a0990ce8090eb88b" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:37.572551 9757 raft_consensus.cc:2454] T 1d182ce2c5084082a0990ce8090eb88b P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 01:59:37.572611 7034 tablet_copy_source_session.cc:205] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.572918 9560 leader_election.cc:302] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:37.573168 9970 raft_consensus.cc:2781] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:37.573232 9970 raft_consensus.cc:482] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:37.573282 9970 raft_consensus.cc:3037] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:37.574311 9970 raft_consensus.cc:504] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:37.574402 10365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ddc47f7144b4cfdbd9a496b5c7562e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.574739 9970 leader_election.cc:288] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Requested vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:37.575264 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d182ce2c5084082a0990ce8090eb88b" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:37.575379 9757 raft_consensus.cc:3037] T 1d182ce2c5084082a0990ce8090eb88b P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:37.576242 9757 raft_consensus.cc:2454] T 1d182ce2c5084082a0990ce8090eb88b P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 01:59:37.576622 9560 leader_election.cc:302] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:37.576834 9970 raft_consensus.cc:2781] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:37.576840 10365 tablet_copy_client.cc:783] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.577011 10365 tablet_copy_client.cc:647] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.577020 9970 raft_consensus.cc:686] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 LEADER]: Becoming Leader. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Running, Role: LEADER
I20240503 01:59:37.577308 9970 consensus_queue.cc:237] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:37.582952 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d182ce2c5084082a0990ce8090eb88b" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:37.583130 9363 raft_consensus.cc:2454] T 1d182ce2c5084082a0990ce8090eb88b P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 01:59:37.583307 9362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d182ce2c5084082a0990ce8090eb88b" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:37.583320 10365 tablet_copy_client.cc:523] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.583379 9362 raft_consensus.cc:3037] T 1d182ce2c5084082a0990ce8090eb88b P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:37.584370 9362 raft_consensus.cc:2454] T 1d182ce2c5084082a0990ce8090eb88b P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 01:59:37.585070 10365 tablet_bootstrap.cc:492] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:37.586391 6740 catalog_manager.cc:5484] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:37.592483 10350 tablet_bootstrap.cc:492] T e811662836164c3fb29a1b12ace7eb41 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:37.592924 10350 tablet_bootstrap.cc:492] T e811662836164c3fb29a1b12ace7eb41 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap complete.
I20240503 01:59:37.593277 10350 ts_tablet_manager.cc:1367] T e811662836164c3fb29a1b12ace7eb41 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.036s user 0.034s sys 0.000s
I20240503 01:59:37.593920 10350 raft_consensus.cc:348] T e811662836164c3fb29a1b12ace7eb41 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:37.594169 10350 raft_consensus.cc:729] T e811662836164c3fb29a1b12ace7eb41 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:37.594398 10350 consensus_queue.cc:260] T e811662836164c3fb29a1b12ace7eb41 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:37.595441 10350 ts_tablet_manager.cc:1398] T e811662836164c3fb29a1b12ace7eb41 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:37.596401 7295 tablet_copy_service.cc:342] P 1d0e738c34af40fa9950c6f086619b3a: Request end of tablet copy session bb5364d730cb4489a0bfc6c270b7450a-e811662836164c3fb29a1b12ace7eb41 received from {username='slave'} at 127.0.99.134:48821
I20240503 01:59:37.596503 7295 tablet_copy_service.cc:433] P 1d0e738c34af40fa9950c6f086619b3a: ending tablet copy session bb5364d730cb4489a0bfc6c270b7450a-e811662836164c3fb29a1b12ace7eb41 on tablet e811662836164c3fb29a1b12ace7eb41 with peer bb5364d730cb4489a0bfc6c270b7450a
I20240503 01:59:37.602820 10342 tablet_bootstrap.cc:492] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:37.603248 10342 tablet_bootstrap.cc:492] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap complete.
I20240503 01:59:37.603503 10342 ts_tablet_manager.cc:1367] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.046s user 0.026s sys 0.006s
I20240503 01:59:37.603986 10342 raft_consensus.cc:348] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:37.604131 10342 raft_consensus.cc:729] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:37.604317 10342 consensus_queue.cc:260] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:37.605542 10342 ts_tablet_manager.cc:1398] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:37.606168 7295 tablet_copy_service.cc:342] P 1d0e738c34af40fa9950c6f086619b3a: Request end of tablet copy session b7062fbbc2b54fff9e99ec4ea5562c76-580e6b4d2b2f4f79b8c2af5e75a3c75f received from {username='slave'} at 127.0.99.135:51921
I20240503 01:59:37.606251 7295 tablet_copy_service.cc:433] P 1d0e738c34af40fa9950c6f086619b3a: ending tablet copy session b7062fbbc2b54fff9e99ec4ea5562c76-580e6b4d2b2f4f79b8c2af5e75a3c75f on tablet 580e6b4d2b2f4f79b8c2af5e75a3c75f with peer b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 01:59:37.608038 7275 consensus_queue.cc:237] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:37.609421 6881 raft_consensus.cc:1261] T 4600418bf64c4a0d962831c6924c46e1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.609596 7145 raft_consensus.cc:1261] T 4600418bf64c4a0d962831c6924c46e1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.609858 9996 consensus_queue.cc:1035] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:37.610239 7210 consensus_peers.cc:485] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): Couldn't send request to peer db1a0000aaa346f79c0aa4b6c5105191. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4600418bf64c4a0d962831c6924c46e1. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.610674 9547 consensus_queue.cc:1035] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.613049 9996 raft_consensus.cc:2932] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:37.613770 7145 raft_consensus.cc:2932] T 4600418bf64c4a0d962831c6924c46e1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:37.617832 6740 catalog_manager.cc:5484] T 4600418bf64c4a0d962831c6924c46e1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:37.618728 6881 raft_consensus.cc:2932] T 4600418bf64c4a0d962831c6924c46e1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:37.624524 10365 tablet_bootstrap.cc:492] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:37.624891 10365 tablet_bootstrap.cc:492] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap complete.
I20240503 01:59:37.625088 10365 ts_tablet_manager.cc:1367] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.040s user 0.022s sys 0.008s
I20240503 01:59:37.625584 10365 raft_consensus.cc:348] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:37.625722 10365 raft_consensus.cc:729] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:37.625896 10365 consensus_queue.cc:260] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:37.626190 10365 ts_tablet_manager.cc:1398] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 01:59:37.626924 7034 tablet_copy_service.cc:342] P 63331f455be049c1bdd6c23c59ac1bef: Request end of tablet copy session db1a0000aaa346f79c0aa4b6c5105191-5ddc47f7144b4cfdbd9a496b5c7562e8 received from {username='slave'} at 127.0.99.136:38763
I20240503 01:59:37.627024 7034 tablet_copy_service.cc:433] P 63331f455be049c1bdd6c23c59ac1bef: ending tablet copy session db1a0000aaa346f79c0aa4b6c5105191-5ddc47f7144b4cfdbd9a496b5c7562e8 on tablet 5ddc47f7144b4cfdbd9a496b5c7562e8 with peer db1a0000aaa346f79c0aa4b6c5105191
I20240503 01:59:37.640926 10350 ts_tablet_manager.cc:902] T 836882aa1fac4721be3cf096540eedc4 P bb5364d730cb4489a0bfc6c270b7450a: Initiating tablet copy from peer 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:37.641451 10350 tablet_copy_client.cc:308] T 836882aa1fac4721be3cf096540eedc4 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36271
I20240503 01:59:37.641928 7295 tablet_copy_service.cc:140] P 1d0e738c34af40fa9950c6f086619b3a: Received BeginTabletCopySession request for tablet 836882aa1fac4721be3cf096540eedc4 from peer bb5364d730cb4489a0bfc6c270b7450a ({username='slave'} at 127.0.99.134:48821)
I20240503 01:59:37.642021 7295 tablet_copy_service.cc:161] P 1d0e738c34af40fa9950c6f086619b3a: Beginning new tablet copy session on tablet 836882aa1fac4721be3cf096540eedc4 from peer bb5364d730cb4489a0bfc6c270b7450a at {username='slave'} at 127.0.99.134:48821: session id = bb5364d730cb4489a0bfc6c270b7450a-836882aa1fac4721be3cf096540eedc4
I20240503 01:59:37.642613 7274 consensus_queue.cc:237] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:37.642797 7295 tablet_copy_source_session.cc:205] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.643666 10350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 836882aa1fac4721be3cf096540eedc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.645498 10350 tablet_copy_client.cc:783] T 836882aa1fac4721be3cf096540eedc4 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.645684 10350 tablet_copy_client.cc:647] T 836882aa1fac4721be3cf096540eedc4 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.646075 6741 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:37.646512 7143 raft_consensus.cc:1261] T 6d4480c8de204f5397dbdcff91db97f6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.647096 10389 consensus_queue.cc:1035] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:37.647886 10350 tablet_copy_client.cc:523] T 836882aa1fac4721be3cf096540eedc4 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.648423 7013 raft_consensus.cc:1261] T 6d4480c8de204f5397dbdcff91db97f6 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:37.649200 9547 consensus_queue.cc:1035] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:37.649556 7211 consensus_peers.cc:485] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d4480c8de204f5397dbdcff91db97f6. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:37.649575 10350 tablet_bootstrap.cc:492] T 836882aa1fac4721be3cf096540eedc4 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:37.652616 9547 raft_consensus.cc:2932] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:37.652920 10360 tablet.cc:1722] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:37.653216 10360 tablet.cc:1620] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:37.653100 7014 raft_consensus.cc:2932] T 6d4480c8de204f5397dbdcff91db97f6 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:37.655226 7143 raft_consensus.cc:2932] T 6d4480c8de204f5397dbdcff91db97f6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:37.656296 6741 catalog_manager.cc:5484] T 6d4480c8de204f5397dbdcff91db97f6 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:37.658432 9914 tablet.cc:1722] T 9aa702eb78ae401f825c945567e8ac3e P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:37.658921 9914 tablet.cc:1620] T 9aa702eb78ae401f825c945567e8ac3e P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 01:59:37.660637 10054 tablet.cc:1722] T 9aa702eb78ae401f825c945567e8ac3e P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:37.660998 10054 tablet.cc:1620] T 9aa702eb78ae401f825c945567e8ac3e P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:37.663003 6738 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=7d48947bbe7d49fc92bed1b29c44a9fe] alter complete (version 2)
I20240503 01:59:37.681530 10350 tablet_bootstrap.cc:492] T 836882aa1fac4721be3cf096540eedc4 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:37.681890 10350 tablet_bootstrap.cc:492] T 836882aa1fac4721be3cf096540eedc4 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap complete.
I20240503 01:59:37.682106 10350 ts_tablet_manager.cc:1367] T 836882aa1fac4721be3cf096540eedc4 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.033s user 0.021s sys 0.000s
I20240503 01:59:37.682639 10350 raft_consensus.cc:348] T 836882aa1fac4721be3cf096540eedc4 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:37.682749 10350 raft_consensus.cc:729] T 836882aa1fac4721be3cf096540eedc4 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:37.682871 10350 consensus_queue.cc:260] T 836882aa1fac4721be3cf096540eedc4 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:37.683140 10350 ts_tablet_manager.cc:1398] T 836882aa1fac4721be3cf096540eedc4 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 01:59:37.683673 7295 tablet_copy_service.cc:342] P 1d0e738c34af40fa9950c6f086619b3a: Request end of tablet copy session bb5364d730cb4489a0bfc6c270b7450a-836882aa1fac4721be3cf096540eedc4 received from {username='slave'} at 127.0.99.134:48821
I20240503 01:59:37.683763 7295 tablet_copy_service.cc:433] P 1d0e738c34af40fa9950c6f086619b3a: ending tablet copy session bb5364d730cb4489a0bfc6c270b7450a-836882aa1fac4721be3cf096540eedc4 on tablet 836882aa1fac4721be3cf096540eedc4 with peer bb5364d730cb4489a0bfc6c270b7450a
I20240503 01:59:37.711294 10365 ts_tablet_manager.cc:902] T c6833c997d284063955d0b9b09ccb3b7 P db1a0000aaa346f79c0aa4b6c5105191: Initiating tablet copy from peer 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:37.711795 10365 tablet_copy_client.cc:308] T c6833c997d284063955d0b9b09ccb3b7 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36271
I20240503 01:59:37.715770 7295 tablet_copy_service.cc:140] P 1d0e738c34af40fa9950c6f086619b3a: Received BeginTabletCopySession request for tablet c6833c997d284063955d0b9b09ccb3b7 from peer db1a0000aaa346f79c0aa4b6c5105191 ({username='slave'} at 127.0.99.136:38163)
I20240503 01:59:37.715898 7295 tablet_copy_service.cc:161] P 1d0e738c34af40fa9950c6f086619b3a: Beginning new tablet copy session on tablet c6833c997d284063955d0b9b09ccb3b7 from peer db1a0000aaa346f79c0aa4b6c5105191 at {username='slave'} at 127.0.99.136:38163: session id = db1a0000aaa346f79c0aa4b6c5105191-c6833c997d284063955d0b9b09ccb3b7
I20240503 01:59:37.716922 7295 tablet_copy_source_session.cc:205] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.717963 10292 ts_tablet_manager.cc:902] T c35ecf8890b64caaac7e5ea9f6e8f989 P 12e8c683db054df3b75c6285b98a5ad8: Initiating tablet copy from peer 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:37.718468 10292 tablet_copy_client.cc:308] T c35ecf8890b64caaac7e5ea9f6e8f989 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:37231
I20240503 01:59:37.719249 7034 tablet_copy_service.cc:140] P 63331f455be049c1bdd6c23c59ac1bef: Received BeginTabletCopySession request for tablet c35ecf8890b64caaac7e5ea9f6e8f989 from peer 12e8c683db054df3b75c6285b98a5ad8 ({username='slave'} at 127.0.99.133:50187)
I20240503 01:59:37.719383 7034 tablet_copy_service.cc:161] P 63331f455be049c1bdd6c23c59ac1bef: Beginning new tablet copy session on tablet c35ecf8890b64caaac7e5ea9f6e8f989 from peer 12e8c683db054df3b75c6285b98a5ad8 at {username='slave'} at 127.0.99.133:50187: session id = 12e8c683db054df3b75c6285b98a5ad8-c35ecf8890b64caaac7e5ea9f6e8f989
I20240503 01:59:37.718498 10365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6833c997d284063955d0b9b09ccb3b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.721585 7034 tablet_copy_source_session.cc:205] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.722930 10292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35ecf8890b64caaac7e5ea9f6e8f989. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.725502 10292 tablet_copy_client.cc:783] T c35ecf8890b64caaac7e5ea9f6e8f989 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.725734 10292 tablet_copy_client.cc:647] T c35ecf8890b64caaac7e5ea9f6e8f989 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.729667 10292 tablet_copy_client.cc:523] T c35ecf8890b64caaac7e5ea9f6e8f989 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.732455 10292 tablet_bootstrap.cc:492] T c35ecf8890b64caaac7e5ea9f6e8f989 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:37.734349 10365 tablet_copy_client.cc:783] T c6833c997d284063955d0b9b09ccb3b7 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.734663 10365 tablet_copy_client.cc:647] T c6833c997d284063955d0b9b09ccb3b7 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.742887 10365 tablet_copy_client.cc:523] T c6833c997d284063955d0b9b09ccb3b7 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.743080 10342 ts_tablet_manager.cc:902] T 5cb5072e91534671ad3b4dd85f93fa5b P b7062fbbc2b54fff9e99ec4ea5562c76: Initiating tablet copy from peer 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:37.743568 10342 tablet_copy_client.cc:308] T 5cb5072e91534671ad3b4dd85f93fa5b P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:37231
I20240503 01:59:37.744419 10365 tablet_bootstrap.cc:492] T c6833c997d284063955d0b9b09ccb3b7 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:37.752408 7034 tablet_copy_service.cc:140] P 63331f455be049c1bdd6c23c59ac1bef: Received BeginTabletCopySession request for tablet 5cb5072e91534671ad3b4dd85f93fa5b from peer b7062fbbc2b54fff9e99ec4ea5562c76 ({username='slave'} at 127.0.99.135:36345)
I20240503 01:59:37.752542 7034 tablet_copy_service.cc:161] P 63331f455be049c1bdd6c23c59ac1bef: Beginning new tablet copy session on tablet 5cb5072e91534671ad3b4dd85f93fa5b from peer b7062fbbc2b54fff9e99ec4ea5562c76 at {username='slave'} at 127.0.99.135:36345: session id = b7062fbbc2b54fff9e99ec4ea5562c76-5cb5072e91534671ad3b4dd85f93fa5b
I20240503 01:59:37.753623 7034 tablet_copy_source_session.cc:205] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.757510 10342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cb5072e91534671ad3b4dd85f93fa5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.760376 10342 tablet_copy_client.cc:783] T 5cb5072e91534671ad3b4dd85f93fa5b P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.762552 10342 tablet_copy_client.cc:647] T 5cb5072e91534671ad3b4dd85f93fa5b P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.765447 10342 tablet_copy_client.cc:523] T 5cb5072e91534671ad3b4dd85f93fa5b P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.767838 10342 tablet_bootstrap.cc:492] T 5cb5072e91534671ad3b4dd85f93fa5b P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:37.770682 6738 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:37.779908 10360 tablet.cc:1722] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:37.782272 10054 tablet.cc:1722] T 9aa702eb78ae401f825c945567e8ac3e P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:37.783172 10292 tablet_bootstrap.cc:492] T c35ecf8890b64caaac7e5ea9f6e8f989 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:37.783459 10292 tablet_bootstrap.cc:492] T c35ecf8890b64caaac7e5ea9f6e8f989 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap complete.
I20240503 01:59:37.783641 10292 ts_tablet_manager.cc:1367] T c35ecf8890b64caaac7e5ea9f6e8f989 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.051s user 0.028s sys 0.000s
I20240503 01:59:37.784096 10292 raft_consensus.cc:348] T c35ecf8890b64caaac7e5ea9f6e8f989 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:37.784227 10292 raft_consensus.cc:729] T c35ecf8890b64caaac7e5ea9f6e8f989 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:37.784396 10292 consensus_queue.cc:260] T c35ecf8890b64caaac7e5ea9f6e8f989 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:37.784868 10292 ts_tablet_manager.cc:1398] T c35ecf8890b64caaac7e5ea9f6e8f989 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:37.785590 7034 tablet_copy_service.cc:342] P 63331f455be049c1bdd6c23c59ac1bef: Request end of tablet copy session 12e8c683db054df3b75c6285b98a5ad8-c35ecf8890b64caaac7e5ea9f6e8f989 received from {username='slave'} at 127.0.99.133:50187
I20240503 01:59:37.785707 7034 tablet_copy_service.cc:433] P 63331f455be049c1bdd6c23c59ac1bef: ending tablet copy session 12e8c683db054df3b75c6285b98a5ad8-c35ecf8890b64caaac7e5ea9f6e8f989 on tablet c35ecf8890b64caaac7e5ea9f6e8f989 with peer 12e8c683db054df3b75c6285b98a5ad8
I20240503 01:59:37.791344 6740 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240503 01:59:37.791503 6740 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240503 01:59:37.792281 9914 tablet.cc:1722] T 9aa702eb78ae401f825c945567e8ac3e P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:37.793259 10365 tablet_bootstrap.cc:492] T c6833c997d284063955d0b9b09ccb3b7 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:37.793553 10365 tablet_bootstrap.cc:492] T c6833c997d284063955d0b9b09ccb3b7 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap complete.
I20240503 01:59:37.793732 10365 ts_tablet_manager.cc:1367] T c6833c997d284063955d0b9b09ccb3b7 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.049s user 0.023s sys 0.001s
I20240503 01:59:37.794222 10365 raft_consensus.cc:348] T c6833c997d284063955d0b9b09ccb3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:37.794328 10365 raft_consensus.cc:729] T c6833c997d284063955d0b9b09ccb3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:37.794451 10365 consensus_queue.cc:260] T c6833c997d284063955d0b9b09ccb3b7 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:37.794865 10292 ts_tablet_manager.cc:902] T a5e23eebc629494cbb8262bc123a720a P 12e8c683db054df3b75c6285b98a5ad8: Initiating tablet copy from peer 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:37.795328 10292 tablet_copy_client.cc:308] T a5e23eebc629494cbb8262bc123a720a P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36271
I20240503 01:59:37.795797 10365 ts_tablet_manager.cc:1398] T c6833c997d284063955d0b9b09ccb3b7 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 01:59:37.797405 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 7343f90f35f1419ab7e1da2fc80f6881
I20240503 01:59:37.797781 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet e24b3cae166f459888fb8b2aef96f942
I20240503 01:59:37.798012 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 4 replicas of tablet 5ddc47f7144b4cfdbd9a496b5c7562e8
I20240503 01:59:37.798084 6863 tablet_service.cc:1514] Processing DeleteTablet for tablet 7343f90f35f1419ab7e1da2fc80f6881 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:37 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:37.798290 9955 tablet_replica.cc:330] T 7343f90f35f1419ab7e1da2fc80f6881 P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:37.798415 9955 raft_consensus.cc:2229] T 7343f90f35f1419ab7e1da2fc80f6881 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:37.798568 9955 raft_consensus.cc:2258] T 7343f90f35f1419ab7e1da2fc80f6881 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:37.799131 9955 ts_tablet_manager.cc:1875] T 7343f90f35f1419ab7e1da2fc80f6881 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:37.799165 7256 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ddc47f7144b4cfdbd9a496b5c7562e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:37 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:37.799384 9736 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ddc47f7144b4cfdbd9a496b5c7562e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:37 UTC) from {username='slave'} at 127.0.0.1:48152
I20240503 01:59:37.799662 10469 tablet_replica.cc:330] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P db1a0000aaa346f79c0aa4b6c5105191: stopping tablet replica
I20240503 01:59:37.799787 10469 raft_consensus.cc:2229] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:37.799908 10469 raft_consensus.cc:2258] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:37.801507 7296 tablet_copy_service.cc:140] P 1d0e738c34af40fa9950c6f086619b3a: Received BeginTabletCopySession request for tablet a5e23eebc629494cbb8262bc123a720a from peer 12e8c683db054df3b75c6285b98a5ad8 ({username='slave'} at 127.0.99.133:37177)
I20240503 01:59:37.801605 7296 tablet_copy_service.cc:161] P 1d0e738c34af40fa9950c6f086619b3a: Beginning new tablet copy session on tablet a5e23eebc629494cbb8262bc123a720a from peer 12e8c683db054df3b75c6285b98a5ad8 at {username='slave'} at 127.0.99.133:37177: session id = 12e8c683db054df3b75c6285b98a5ad8-a5e23eebc629494cbb8262bc123a720a
I20240503 01:59:37.802135 9955 ts_tablet_manager.cc:1888] T 7343f90f35f1419ab7e1da2fc80f6881 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:37.802201 9955 log.cc:1196] T 7343f90f35f1419ab7e1da2fc80f6881 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/7343f90f35f1419ab7e1da2fc80f6881
I20240503 01:59:37.802287 7296 tablet_copy_source_session.cc:205] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.802444 9955 ts_tablet_manager.cc:1909] T 7343f90f35f1419ab7e1da2fc80f6881 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:37.802659 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ddc47f7144b4cfdbd9a496b5c7562e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:37 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:37.802855 7254 tablet_service.cc:1514] Processing DeleteTablet for tablet 7343f90f35f1419ab7e1da2fc80f6881 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:37 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:37.802982 7255 tablet_service.cc:1514] Processing DeleteTablet for tablet e24b3cae166f459888fb8b2aef96f942 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:37 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:37.803102 7124 tablet_service.cc:1514] Processing DeleteTablet for tablet e24b3cae166f459888fb8b2aef96f942 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:37 UTC) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:37.803238 10292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5e23eebc629494cbb8262bc123a720a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.803602 6863 tablet_service.cc:1514] Processing DeleteTablet for tablet e24b3cae166f459888fb8b2aef96f942 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:37 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:37.804488 9955 tablet_replica.cc:330] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:37.804634 9955 raft_consensus.cc:2229] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:37.804762 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 7343f90f35f1419ab7e1da2fc80f6881 (table rebalance_test_table_6 [id=d4b0f87e5f7a49e897a3904f2d2e871c]) successfully deleted
I20240503 01:59:37.804813 9955 raft_consensus.cc:2258] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:37.805039 7295 tablet_copy_service.cc:342] P 1d0e738c34af40fa9950c6f086619b3a: Request end of tablet copy session db1a0000aaa346f79c0aa4b6c5105191-c6833c997d284063955d0b9b09ccb3b7 received from {username='slave'} at 127.0.99.136:38163
I20240503 01:59:37.805110 7295 tablet_copy_service.cc:433] P 1d0e738c34af40fa9950c6f086619b3a: ending tablet copy session db1a0000aaa346f79c0aa4b6c5105191-c6833c997d284063955d0b9b09ccb3b7 on tablet c6833c997d284063955d0b9b09ccb3b7 with peer db1a0000aaa346f79c0aa4b6c5105191
I20240503 01:59:37.805302 6992 tablet_service.cc:1514] Processing DeleteTablet for tablet 7343f90f35f1419ab7e1da2fc80f6881 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:37 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:37.805563 10292 tablet_copy_client.cc:783] T a5e23eebc629494cbb8262bc123a720a P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.805734 10292 tablet_copy_client.cc:647] T a5e23eebc629494cbb8262bc123a720a P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.805912 6992 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ddc47f7144b4cfdbd9a496b5c7562e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:37 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:37.806998 9955 ts_tablet_manager.cc:1875] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:37.809247 9955 ts_tablet_manager.cc:1888] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:37.809315 9955 log.cc:1196] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/5ddc47f7144b4cfdbd9a496b5c7562e8
I20240503 01:59:37.809584 9955 ts_tablet_manager.cc:1909] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:37.809222 6737 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=7d48947bbe7d49fc92bed1b29c44a9fe] alter complete (version 3)
I20240503 01:59:37.810371 9955 tablet_replica.cc:330] T e24b3cae166f459888fb8b2aef96f942 P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:37.810395 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 5ddc47f7144b4cfdbd9a496b5c7562e8 (table rebalance_test_table_6 [id=d4b0f87e5f7a49e897a3904f2d2e871c]) successfully deleted
I20240503 01:59:37.810504 9955 raft_consensus.cc:2229] T e24b3cae166f459888fb8b2aef96f942 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:37.810668 9955 raft_consensus.cc:2258] T e24b3cae166f459888fb8b2aef96f942 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:37.811156 9955 ts_tablet_manager.cc:1875] T e24b3cae166f459888fb8b2aef96f942 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:37.812064 10292 tablet_copy_client.cc:523] T a5e23eebc629494cbb8262bc123a720a P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.812867 10469 ts_tablet_manager.cc:1875] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P db1a0000aaa346f79c0aa4b6c5105191: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:37.813509 10292 tablet_bootstrap.cc:492] T a5e23eebc629494cbb8262bc123a720a P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:37.814710 10470 tablet_replica.cc:330] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:37.814879 10470 raft_consensus.cc:2229] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:37.815116 10470 raft_consensus.cc:2258] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:37.816022 10469 ts_tablet_manager.cc:1888] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P db1a0000aaa346f79c0aa4b6c5105191: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:37.816112 10469 log.cc:1196] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P db1a0000aaa346f79c0aa4b6c5105191: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/5ddc47f7144b4cfdbd9a496b5c7562e8
I20240503 01:59:37.816385 10469 ts_tablet_manager.cc:1909] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P db1a0000aaa346f79c0aa4b6c5105191: Deleting consensus metadata
I20240503 01:59:37.816888 10470 ts_tablet_manager.cc:1875] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:37.817914 10365 ts_tablet_manager.cc:902] T dec7e3036d384f5db51f50ba93acbbc6 P db1a0000aaa346f79c0aa4b6c5105191: Initiating tablet copy from peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:37.818344 10365 tablet_copy_client.cc:308] T dec7e3036d384f5db51f50ba93acbbc6 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:42943
I20240503 01:59:37.819144 6726 catalog_manager.cc:4830] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): tablet 5ddc47f7144b4cfdbd9a496b5c7562e8 (table rebalance_test_table_6 [id=d4b0f87e5f7a49e897a3904f2d2e871c]) successfully deleted
I20240503 01:59:37.821875 10470 ts_tablet_manager.cc:1888] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:37.822185 10470 log.cc:1196] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/e24b3cae166f459888fb8b2aef96f942
I20240503 01:59:37.822559 10470 ts_tablet_manager.cc:1909] T e24b3cae166f459888fb8b2aef96f942 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting consensus metadata
I20240503 01:59:37.822861 9955 ts_tablet_manager.cc:1888] T e24b3cae166f459888fb8b2aef96f942 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:37.822948 9955 log.cc:1196] T e24b3cae166f459888fb8b2aef96f942 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/e24b3cae166f459888fb8b2aef96f942
I20240503 01:59:37.823220 9955 ts_tablet_manager.cc:1909] T e24b3cae166f459888fb8b2aef96f942 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:37.823535 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet e24b3cae166f459888fb8b2aef96f942 (table rebalance_test_table_6 [id=d4b0f87e5f7a49e897a3904f2d2e871c]) successfully deleted
I20240503 01:59:37.823840 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet e24b3cae166f459888fb8b2aef96f942 (table rebalance_test_table_6 [id=d4b0f87e5f7a49e897a3904f2d2e871c]) successfully deleted
I20240503 01:59:37.823958 10471 tablet_replica.cc:330] T 7343f90f35f1419ab7e1da2fc80f6881 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:37.824143 10471 raft_consensus.cc:2229] T 7343f90f35f1419ab7e1da2fc80f6881 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:37.824270 10468 tablet_replica.cc:330] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:37.824326 10471 raft_consensus.cc:2258] T 7343f90f35f1419ab7e1da2fc80f6881 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:37.824707 10468 raft_consensus.cc:2229] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:37.824862 10471 ts_tablet_manager.cc:1875] T 7343f90f35f1419ab7e1da2fc80f6881 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:37.825095 10468 raft_consensus.cc:2258] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:37.826936 10468 ts_tablet_manager.cc:1875] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:37.826999 6903 tablet_copy_service.cc:140] P 5dcc33cc621a4cbbae155f3230fb7c46: Received BeginTabletCopySession request for tablet dec7e3036d384f5db51f50ba93acbbc6 from peer db1a0000aaa346f79c0aa4b6c5105191 ({username='slave'} at 127.0.99.136:40711)
I20240503 01:59:37.827142 6903 tablet_copy_service.cc:161] P 5dcc33cc621a4cbbae155f3230fb7c46: Beginning new tablet copy session on tablet dec7e3036d384f5db51f50ba93acbbc6 from peer db1a0000aaa346f79c0aa4b6c5105191 at {username='slave'} at 127.0.99.136:40711: session id = db1a0000aaa346f79c0aa4b6c5105191-dec7e3036d384f5db51f50ba93acbbc6
I20240503 01:59:37.828574 6903 tablet_copy_source_session.cc:205] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.830062 10471 ts_tablet_manager.cc:1888] T 7343f90f35f1419ab7e1da2fc80f6881 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:37.830168 10471 log.cc:1196] T 7343f90f35f1419ab7e1da2fc80f6881 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/7343f90f35f1419ab7e1da2fc80f6881
I20240503 01:59:37.830229 10365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dec7e3036d384f5db51f50ba93acbbc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.830744 10468 ts_tablet_manager.cc:1888] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:37.830850 10468 log.cc:1196] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/5ddc47f7144b4cfdbd9a496b5c7562e8
I20240503 01:59:37.831125 10468 ts_tablet_manager.cc:1909] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:37.831902 10468 tablet_replica.cc:330] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:37.832041 10468 raft_consensus.cc:2229] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:37.832214 10468 raft_consensus.cc:2258] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:37.832902 10468 ts_tablet_manager.cc:1875] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:37.833115 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet 5ddc47f7144b4cfdbd9a496b5c7562e8 (table rebalance_test_table_6 [id=d4b0f87e5f7a49e897a3904f2d2e871c]) successfully deleted
I20240503 01:59:37.833223 10365 tablet_copy_client.cc:783] T dec7e3036d384f5db51f50ba93acbbc6 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.833453 10365 tablet_copy_client.cc:647] T dec7e3036d384f5db51f50ba93acbbc6 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.836493 10468 ts_tablet_manager.cc:1888] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:37.836740 10468 log.cc:1196] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/7343f90f35f1419ab7e1da2fc80f6881
I20240503 01:59:37.837190 10468 ts_tablet_manager.cc:1909] T 7343f90f35f1419ab7e1da2fc80f6881 P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:37.837358 10078 consensus_queue.cc:1035] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:37.839188 10365 tablet_copy_client.cc:523] T dec7e3036d384f5db51f50ba93acbbc6 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.840818 10365 tablet_bootstrap.cc:492] T dec7e3036d384f5db51f50ba93acbbc6 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:37.842005 10471 ts_tablet_manager.cc:1909] T 7343f90f35f1419ab7e1da2fc80f6881 P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:37.842856 10471 tablet_replica.cc:330] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:37.842936 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet 7343f90f35f1419ab7e1da2fc80f6881 (table rebalance_test_table_6 [id=d4b0f87e5f7a49e897a3904f2d2e871c]) successfully deleted
I20240503 01:59:37.843103 10471 raft_consensus.cc:2229] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:37.843423 10471 raft_consensus.cc:2258] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:37.844836 10468 tablet_replica.cc:330] T e24b3cae166f459888fb8b2aef96f942 P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:37.844935 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet 7343f90f35f1419ab7e1da2fc80f6881 (table rebalance_test_table_6 [id=d4b0f87e5f7a49e897a3904f2d2e871c]) successfully deleted
I20240503 01:59:37.845093 10468 raft_consensus.cc:2229] T e24b3cae166f459888fb8b2aef96f942 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:37.845281 10468 raft_consensus.cc:2258] T e24b3cae166f459888fb8b2aef96f942 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:37.845523 10471 ts_tablet_manager.cc:1875] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:37.845981 10468 ts_tablet_manager.cc:1875] T e24b3cae166f459888fb8b2aef96f942 P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:37.849418 10468 ts_tablet_manager.cc:1888] T e24b3cae166f459888fb8b2aef96f942 P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:37.849553 10468 log.cc:1196] T e24b3cae166f459888fb8b2aef96f942 P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/e24b3cae166f459888fb8b2aef96f942
I20240503 01:59:37.849855 10468 ts_tablet_manager.cc:1909] T e24b3cae166f459888fb8b2aef96f942 P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:37.851898 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet e24b3cae166f459888fb8b2aef96f942 (table rebalance_test_table_6 [id=d4b0f87e5f7a49e897a3904f2d2e871c]) successfully deleted
I20240503 01:59:37.854792 9913 consensus_queue.cc:1035] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:37.855206 10471 ts_tablet_manager.cc:1888] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:37.855297 10471 log.cc:1196] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/5ddc47f7144b4cfdbd9a496b5c7562e8
I20240503 01:59:37.855592 10471 ts_tablet_manager.cc:1909] T 5ddc47f7144b4cfdbd9a496b5c7562e8 P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:37.857622 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet 5ddc47f7144b4cfdbd9a496b5c7562e8 (table rebalance_test_table_6 [id=d4b0f87e5f7a49e897a3904f2d2e871c]) successfully deleted
I20240503 01:59:37.865322 9343 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:37.883539 6863 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:37.891383 6992 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 01:59:37.903366 7124 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:37.903678 7255 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 01:59:37.912078 9494 raft_consensus.cc:1203] T e811662836164c3fb29a1b12ace7eb41 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:37.915774 9362 raft_consensus.cc:1203] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:37.917255 6738 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:37.917411 6738 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:37.924532 6738 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 9aa702eb78ae401f825c945567e8ac3e
I20240503 01:59:37.925351 9343 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aa702eb78ae401f825c945567e8ac3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:37 UTC) from {username='slave'} at 127.0.0.1:36040
I20240503 01:59:37.925383 9736 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aa702eb78ae401f825c945567e8ac3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:37 UTC) from {username='slave'} at 127.0.0.1:48152
I20240503 01:59:37.925570 10469 tablet_replica.cc:330] T 9aa702eb78ae401f825c945567e8ac3e P db1a0000aaa346f79c0aa4b6c5105191: stopping tablet replica
I20240503 01:59:37.925695 10469 raft_consensus.cc:2229] T 9aa702eb78ae401f825c945567e8ac3e P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:37.925806 10469 raft_consensus.cc:2258] T 9aa702eb78ae401f825c945567e8ac3e P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:37.926473 9474 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aa702eb78ae401f825c945567e8ac3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:37 UTC) from {username='slave'} at 127.0.0.1:54678
I20240503 01:59:37.928602 10469 ts_tablet_manager.cc:1875] T 9aa702eb78ae401f825c945567e8ac3e P db1a0000aaa346f79c0aa4b6c5105191: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:37.930958 9625 raft_consensus.cc:1203] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:37.931344 10469 ts_tablet_manager.cc:1888] T 9aa702eb78ae401f825c945567e8ac3e P db1a0000aaa346f79c0aa4b6c5105191: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:37.931412 10469 log.cc:1196] T 9aa702eb78ae401f825c945567e8ac3e P db1a0000aaa346f79c0aa4b6c5105191: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/9aa702eb78ae401f825c945567e8ac3e
I20240503 01:59:37.931682 10469 ts_tablet_manager.cc:1909] T 9aa702eb78ae401f825c945567e8ac3e P db1a0000aaa346f79c0aa4b6c5105191: Deleting consensus metadata
I20240503 01:59:37.933252 6726 catalog_manager.cc:4830] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): tablet 9aa702eb78ae401f825c945567e8ac3e (table rebalancer_dynamic_table_new_name [id=7d48947bbe7d49fc92bed1b29c44a9fe]) successfully deleted
I20240503 01:59:37.936522 9604 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:37.937332 9474 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:37.939163 9736 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:37.941655 10495 tablet_replica.cc:330] T 9aa702eb78ae401f825c945567e8ac3e P 12e8c683db054df3b75c6285b98a5ad8: stopping tablet replica
I20240503 01:59:37.942202 10496 tablet_replica.cc:330] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a: stopping tablet replica
I20240503 01:59:37.942415 10495 raft_consensus.cc:2229] T 9aa702eb78ae401f825c945567e8ac3e P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:37.943133 10495 raft_consensus.cc:2258] T 9aa702eb78ae401f825c945567e8ac3e P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:37.943406 10496 raft_consensus.cc:2229] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:37.944617 10496 raft_consensus.cc:2258] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:37.948149 10342 tablet_bootstrap.cc:492] T 5cb5072e91534671ad3b4dd85f93fa5b P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:37.950022 10342 tablet_bootstrap.cc:492] T 5cb5072e91534671ad3b4dd85f93fa5b P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap complete.
I20240503 01:59:37.951717 10342 ts_tablet_manager.cc:1367] T 5cb5072e91534671ad3b4dd85f93fa5b P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.184s user 0.025s sys 0.005s
I20240503 01:59:37.952296 10342 raft_consensus.cc:348] T 5cb5072e91534671ad3b4dd85f93fa5b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:37.952543 10342 raft_consensus.cc:729] T 5cb5072e91534671ad3b4dd85f93fa5b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:37.953439 10342 consensus_queue.cc:260] T 5cb5072e91534671ad3b4dd85f93fa5b P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:37.953732 10292 tablet_bootstrap.cc:492] T a5e23eebc629494cbb8262bc123a720a P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:37.956154 10342 ts_tablet_manager.cc:1398] T 5cb5072e91534671ad3b4dd85f93fa5b P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:37.953853 10497 ts_tablet_manager.cc:902] T 49b580d648014ce899fee8b3b329c47d P b7062fbbc2b54fff9e99ec4ea5562c76: Initiating tablet copy from peer 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:37.958578 7034 tablet_copy_service.cc:342] P 63331f455be049c1bdd6c23c59ac1bef: Request end of tablet copy session b7062fbbc2b54fff9e99ec4ea5562c76-5cb5072e91534671ad3b4dd85f93fa5b received from {username='slave'} at 127.0.99.135:36345
I20240503 01:59:37.958674 7034 tablet_copy_service.cc:433] P 63331f455be049c1bdd6c23c59ac1bef: ending tablet copy session b7062fbbc2b54fff9e99ec4ea5562c76-5cb5072e91534671ad3b4dd85f93fa5b on tablet 5cb5072e91534671ad3b4dd85f93fa5b with peer b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 01:59:37.959995 10497 tablet_copy_client.cc:308] T 49b580d648014ce899fee8b3b329c47d P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:37231
I20240503 01:59:37.960377 10365 tablet_bootstrap.cc:492] T dec7e3036d384f5db51f50ba93acbbc6 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:37.961108 7034 tablet_copy_service.cc:140] P 63331f455be049c1bdd6c23c59ac1bef: Received BeginTabletCopySession request for tablet 49b580d648014ce899fee8b3b329c47d from peer b7062fbbc2b54fff9e99ec4ea5562c76 ({username='slave'} at 127.0.99.135:36345)
I20240503 01:59:37.961228 7034 tablet_copy_service.cc:161] P 63331f455be049c1bdd6c23c59ac1bef: Beginning new tablet copy session on tablet 49b580d648014ce899fee8b3b329c47d from peer b7062fbbc2b54fff9e99ec4ea5562c76 at {username='slave'} at 127.0.99.135:36345: session id = b7062fbbc2b54fff9e99ec4ea5562c76-49b580d648014ce899fee8b3b329c47d
I20240503 01:59:37.961731 10292 tablet_bootstrap.cc:492] T a5e23eebc629494cbb8262bc123a720a P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap complete.
I20240503 01:59:37.962006 10292 ts_tablet_manager.cc:1367] T a5e23eebc629494cbb8262bc123a720a P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.149s user 0.027s sys 0.004s
I20240503 01:59:37.962164 7034 tablet_copy_source_session.cc:205] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.963120 10292 raft_consensus.cc:348] T a5e23eebc629494cbb8262bc123a720a P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:37.965533 10496 ts_tablet_manager.cc:1875] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:37.965723 10292 raft_consensus.cc:729] T a5e23eebc629494cbb8262bc123a720a P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:37.965961 10292 consensus_queue.cc:260] T a5e23eebc629494cbb8262bc123a720a P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:37.966368 10292 ts_tablet_manager.cc:1398] T a5e23eebc629494cbb8262bc123a720a P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:37.971711 7295 tablet_copy_service.cc:342] P 1d0e738c34af40fa9950c6f086619b3a: Request end of tablet copy session 12e8c683db054df3b75c6285b98a5ad8-a5e23eebc629494cbb8262bc123a720a received from {username='slave'} at 127.0.99.133:37177
I20240503 01:59:37.971818 7295 tablet_copy_service.cc:433] P 1d0e738c34af40fa9950c6f086619b3a: ending tablet copy session 12e8c683db054df3b75c6285b98a5ad8-a5e23eebc629494cbb8262bc123a720a on tablet a5e23eebc629494cbb8262bc123a720a with peer 12e8c683db054df3b75c6285b98a5ad8
I20240503 01:59:37.968096 10495 ts_tablet_manager.cc:1875] T 9aa702eb78ae401f825c945567e8ac3e P 12e8c683db054df3b75c6285b98a5ad8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:37.972307 10365 tablet_bootstrap.cc:492] T dec7e3036d384f5db51f50ba93acbbc6 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap complete.
I20240503 01:59:37.973029 10365 ts_tablet_manager.cc:1367] T dec7e3036d384f5db51f50ba93acbbc6 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.132s user 0.028s sys 0.000s
I20240503 01:59:37.973415 10350 ts_tablet_manager.cc:902] T 8ed2793842464443826f83f60fb6dcd1 P bb5364d730cb4489a0bfc6c270b7450a: Initiating tablet copy from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:37.973635 10365 raft_consensus.cc:348] T dec7e3036d384f5db51f50ba93acbbc6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:37.973748 10365 raft_consensus.cc:729] T dec7e3036d384f5db51f50ba93acbbc6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:37.973860 10350 tablet_copy_client.cc:308] T 8ed2793842464443826f83f60fb6dcd1 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41979
I20240503 01:59:37.974750 10495 ts_tablet_manager.cc:1888] T 9aa702eb78ae401f825c945567e8ac3e P 12e8c683db054df3b75c6285b98a5ad8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:37.974862 10495 log.cc:1196] T 9aa702eb78ae401f825c945567e8ac3e P 12e8c683db054df3b75c6285b98a5ad8: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/9aa702eb78ae401f825c945567e8ac3e
I20240503 01:59:37.974939 10365 consensus_queue.cc:260] T dec7e3036d384f5db51f50ba93acbbc6 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:37.975236 10495 ts_tablet_manager.cc:1909] T 9aa702eb78ae401f825c945567e8ac3e P 12e8c683db054df3b75c6285b98a5ad8: Deleting consensus metadata
I20240503 01:59:37.975978 6727 catalog_manager.cc:4830] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): tablet 9aa702eb78ae401f825c945567e8ac3e (table rebalancer_dynamic_table_new_name [id=7d48947bbe7d49fc92bed1b29c44a9fe]) successfully deleted
I20240503 01:59:37.977082 10365 ts_tablet_manager.cc:1398] T dec7e3036d384f5db51f50ba93acbbc6 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:37.978890 7165 tablet_copy_service.cc:140] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Received BeginTabletCopySession request for tablet 8ed2793842464443826f83f60fb6dcd1 from peer bb5364d730cb4489a0bfc6c270b7450a ({username='slave'} at 127.0.99.134:59853)
I20240503 01:59:37.979032 7165 tablet_copy_service.cc:161] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Beginning new tablet copy session on tablet 8ed2793842464443826f83f60fb6dcd1 from peer bb5364d730cb4489a0bfc6c270b7450a at {username='slave'} at 127.0.99.134:59853: session id = bb5364d730cb4489a0bfc6c270b7450a-8ed2793842464443826f83f60fb6dcd1
I20240503 01:59:37.979519 10496 ts_tablet_manager.cc:1888] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:37.979707 10496 log.cc:1196] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/9aa702eb78ae401f825c945567e8ac3e
I20240503 01:59:37.980083 10496 ts_tablet_manager.cc:1909] T 9aa702eb78ae401f825c945567e8ac3e P bb5364d730cb4489a0bfc6c270b7450a: Deleting consensus metadata
I20240503 01:59:37.980453 7165 tablet_copy_source_session.cc:205] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:37.980715 6903 tablet_copy_service.cc:342] P 5dcc33cc621a4cbbae155f3230fb7c46: Request end of tablet copy session db1a0000aaa346f79c0aa4b6c5105191-dec7e3036d384f5db51f50ba93acbbc6 received from {username='slave'} at 127.0.99.136:40711
I20240503 01:59:37.980821 6903 tablet_copy_service.cc:433] P 5dcc33cc621a4cbbae155f3230fb7c46: ending tablet copy session db1a0000aaa346f79c0aa4b6c5105191-dec7e3036d384f5db51f50ba93acbbc6 on tablet dec7e3036d384f5db51f50ba93acbbc6 with peer db1a0000aaa346f79c0aa4b6c5105191
I20240503 01:59:37.981698 10350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ed2793842464443826f83f60fb6dcd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.982686 6725 catalog_manager.cc:4830] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): tablet 9aa702eb78ae401f825c945567e8ac3e (table rebalancer_dynamic_table_new_name [id=7d48947bbe7d49fc92bed1b29c44a9fe]) successfully deleted
I20240503 01:59:37.984149 10350 tablet_copy_client.cc:783] T 8ed2793842464443826f83f60fb6dcd1 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.984390 10350 tablet_copy_client.cc:647] T 8ed2793842464443826f83f60fb6dcd1 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.985873 10350 tablet_copy_client.cc:523] T 8ed2793842464443826f83f60fb6dcd1 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:37.986945 9970 consensus_queue.cc:1035] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:37.987331 10350 tablet_bootstrap.cc:492] T 8ed2793842464443826f83f60fb6dcd1 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:37.987630 10497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49b580d648014ce899fee8b3b329c47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:37.992448 9970 consensus_queue.cc:1035] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:37.997538 10497 tablet_copy_client.cc:783] T 49b580d648014ce899fee8b3b329c47d P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:37.997787 10497 tablet_copy_client.cc:647] T 49b580d648014ce899fee8b3b329c47d P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:37.999737 10497 tablet_copy_client.cc:523] T 49b580d648014ce899fee8b3b329c47d P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.001226 10497 tablet_bootstrap.cc:492] T 49b580d648014ce899fee8b3b329c47d P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:38.012112 10350 tablet_bootstrap.cc:492] T 8ed2793842464443826f83f60fb6dcd1 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.012530 10350 tablet_bootstrap.cc:492] T 8ed2793842464443826f83f60fb6dcd1 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap complete.
I20240503 01:59:38.012811 10350 ts_tablet_manager.cc:1367] T 8ed2793842464443826f83f60fb6dcd1 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.026s user 0.023s sys 0.002s
I20240503 01:59:38.013396 10350 raft_consensus.cc:348] T 8ed2793842464443826f83f60fb6dcd1 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:38.013615 10350 raft_consensus.cc:729] T 8ed2793842464443826f83f60fb6dcd1 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:38.013844 10350 consensus_queue.cc:260] T 8ed2793842464443826f83f60fb6dcd1 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:38.016065 10350 ts_tablet_manager.cc:1398] T 8ed2793842464443826f83f60fb6dcd1 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:38.016959 9055 consensus_queue.cc:1035] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:38.018126 7165 tablet_copy_service.cc:342] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Request end of tablet copy session bb5364d730cb4489a0bfc6c270b7450a-8ed2793842464443826f83f60fb6dcd1 received from {username='slave'} at 127.0.99.134:59853
I20240503 01:59:38.018230 7165 tablet_copy_service.cc:433] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: ending tablet copy session bb5364d730cb4489a0bfc6c270b7450a-8ed2793842464443826f83f60fb6dcd1 on tablet 8ed2793842464443826f83f60fb6dcd1 with peer bb5364d730cb4489a0bfc6c270b7450a
I20240503 01:59:38.025656 10365 ts_tablet_manager.cc:902] T 4600418bf64c4a0d962831c6924c46e1 P db1a0000aaa346f79c0aa4b6c5105191: Initiating tablet copy from peer 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:38.026347 10365 tablet_copy_client.cc:308] T 4600418bf64c4a0d962831c6924c46e1 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36271
I20240503 01:59:38.026969 7295 tablet_copy_service.cc:140] P 1d0e738c34af40fa9950c6f086619b3a: Received BeginTabletCopySession request for tablet 4600418bf64c4a0d962831c6924c46e1 from peer db1a0000aaa346f79c0aa4b6c5105191 ({username='slave'} at 127.0.99.136:38163)
I20240503 01:59:38.027101 7295 tablet_copy_service.cc:161] P 1d0e738c34af40fa9950c6f086619b3a: Beginning new tablet copy session on tablet 4600418bf64c4a0d962831c6924c46e1 from peer db1a0000aaa346f79c0aa4b6c5105191 at {username='slave'} at 127.0.99.136:38163: session id = db1a0000aaa346f79c0aa4b6c5105191-4600418bf64c4a0d962831c6924c46e1
I20240503 01:59:38.028106 7295 tablet_copy_source_session.cc:205] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.028702 6738 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:38.029556 10079 consensus_queue.cc:1035] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:38.030886 10365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4600418bf64c4a0d962831c6924c46e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.032991 10365 tablet_copy_client.cc:783] T 4600418bf64c4a0d962831c6924c46e1 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.033146 10365 tablet_copy_client.cc:647] T 4600418bf64c4a0d962831c6924c46e1 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.037883 9474 tablet_service.cc:1467] Processing CreateTablet for tablet de80a938a89f471b8b9c0c4b1b4be326 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=70b7076aa6314855b03666d69272e9fd]), partition=
I20240503 01:59:38.037894 9736 tablet_service.cc:1467] Processing CreateTablet for tablet de80a938a89f471b8b9c0c4b1b4be326 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=70b7076aa6314855b03666d69272e9fd]), partition=
I20240503 01:59:38.038134 9474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de80a938a89f471b8b9c0c4b1b4be326. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.038134 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de80a938a89f471b8b9c0c4b1b4be326. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.039002 9604 tablet_service.cc:1467] Processing CreateTablet for tablet de80a938a89f471b8b9c0c4b1b4be326 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=70b7076aa6314855b03666d69272e9fd]), partition=
I20240503 01:59:38.039222 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de80a938a89f471b8b9c0c4b1b4be326. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.041121 10365 tablet_copy_client.cc:523] T 4600418bf64c4a0d962831c6924c46e1 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.042172 10365 tablet_bootstrap.cc:492] T 4600418bf64c4a0d962831c6924c46e1 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:38.042657 10292 ts_tablet_manager.cc:902] T ccbd932e20b44dc5aceb86c86d7fda6b P 12e8c683db054df3b75c6285b98a5ad8: Initiating tablet copy from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:38.043407 10292 tablet_copy_client.cc:308] T ccbd932e20b44dc5aceb86c86d7fda6b P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41979
I20240503 01:59:38.044396 10529 tablet_bootstrap.cc:492] T de80a938a89f471b8b9c0c4b1b4be326 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:38.045820 10529 tablet_bootstrap.cc:654] T de80a938a89f471b8b9c0c4b1b4be326 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:38.047902 10528 tablet_bootstrap.cc:492] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:38.049465 7165 tablet_copy_service.cc:140] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Received BeginTabletCopySession request for tablet ccbd932e20b44dc5aceb86c86d7fda6b from peer 12e8c683db054df3b75c6285b98a5ad8 ({username='slave'} at 127.0.99.133:44539)
I20240503 01:59:38.049612 7165 tablet_copy_service.cc:161] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Beginning new tablet copy session on tablet ccbd932e20b44dc5aceb86c86d7fda6b from peer 12e8c683db054df3b75c6285b98a5ad8 at {username='slave'} at 127.0.99.133:44539: session id = 12e8c683db054df3b75c6285b98a5ad8-ccbd932e20b44dc5aceb86c86d7fda6b
I20240503 01:59:38.050609 7165 tablet_copy_source_session.cc:205] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.048967 10529 tablet_bootstrap.cc:492] T de80a938a89f471b8b9c0c4b1b4be326 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:38.051429 10529 ts_tablet_manager.cc:1367] T de80a938a89f471b8b9c0c4b1b4be326 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:59:38.051748 10292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccbd932e20b44dc5aceb86c86d7fda6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.048184 10527 tablet_bootstrap.cc:492] T de80a938a89f471b8b9c0c4b1b4be326 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:38.054050 10292 tablet_copy_client.cc:783] T ccbd932e20b44dc5aceb86c86d7fda6b P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.053794 10529 raft_consensus.cc:348] T de80a938a89f471b8b9c0c4b1b4be326 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:38.054225 10292 tablet_copy_client.cc:647] T ccbd932e20b44dc5aceb86c86d7fda6b P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.054244 10529 raft_consensus.cc:374] T de80a938a89f471b8b9c0c4b1b4be326 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:38.054280 10529 raft_consensus.cc:729] T de80a938a89f471b8b9c0c4b1b4be326 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:38.055341 10527 tablet_bootstrap.cc:654] T de80a938a89f471b8b9c0c4b1b4be326 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:38.055963 10529 consensus_queue.cc:260] T de80a938a89f471b8b9c0c4b1b4be326 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:38.056455 10528 tablet_bootstrap.cc:654] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:38.056543 10292 tablet_copy_client.cc:523] T ccbd932e20b44dc5aceb86c86d7fda6b P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.057219 10529 ts_tablet_manager.cc:1398] T de80a938a89f471b8b9c0c4b1b4be326 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 01:59:38.057859 10292 tablet_bootstrap.cc:492] T ccbd932e20b44dc5aceb86c86d7fda6b P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:38.059734 10528 tablet_bootstrap.cc:492] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:38.059962 10528 ts_tablet_manager.cc:1367] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 01:59:38.060496 10528 raft_consensus.cc:348] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:38.060737 10528 raft_consensus.cc:374] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:38.060817 10528 raft_consensus.cc:729] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:38.061023 10528 consensus_queue.cc:260] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:38.061239 9757 consensus_queue.cc:237] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:38.061362 10528 ts_tablet_manager.cc:1398] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:38.063879 9494 raft_consensus.cc:1261] T 31b077072738410f8d7c139ad1b5aa9c P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 01:59:38.064232 9693 consensus_peers.cc:485] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): Couldn't send request to peer 5dcc33cc621a4cbbae155f3230fb7c46. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 31b077072738410f8d7c139ad1b5aa9c. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.064371 9973 consensus_queue.cc:1035] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:38.065078 9362 raft_consensus.cc:1261] T 31b077072738410f8d7c139ad1b5aa9c P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:38.065524 9973 consensus_queue.cc:1035] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:38.067364 9973 raft_consensus.cc:2932] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:38.068082 10497 tablet_bootstrap.cc:492] T 49b580d648014ce899fee8b3b329c47d P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.068378 10497 tablet_bootstrap.cc:492] T 49b580d648014ce899fee8b3b329c47d P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap complete.
I20240503 01:59:38.068547 10497 ts_tablet_manager.cc:1367] T 49b580d648014ce899fee8b3b329c47d P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.067s user 0.030s sys 0.000s
I20240503 01:59:38.069751 10497 raft_consensus.cc:348] T 49b580d648014ce899fee8b3b329c47d P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:38.069897 9494 raft_consensus.cc:2932] T 31b077072738410f8d7c139ad1b5aa9c P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:38.070477 9362 raft_consensus.cc:2932] T 31b077072738410f8d7c139ad1b5aa9c P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:38.072396 6737 catalog_manager.cc:5484] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:38.075330 10527 tablet_bootstrap.cc:492] T de80a938a89f471b8b9c0c4b1b4be326 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:38.075426 10527 ts_tablet_manager.cc:1367] T de80a938a89f471b8b9c0c4b1b4be326 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20240503 01:59:38.075909 10527 raft_consensus.cc:348] T de80a938a89f471b8b9c0c4b1b4be326 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:38.076021 10527 raft_consensus.cc:374] T de80a938a89f471b8b9c0c4b1b4be326 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:38.076051 10527 raft_consensus.cc:729] T de80a938a89f471b8b9c0c4b1b4be326 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:38.076164 10527 consensus_queue.cc:260] T de80a938a89f471b8b9c0c4b1b4be326 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:38.076577 10527 ts_tablet_manager.cc:1398] T de80a938a89f471b8b9c0c4b1b4be326 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:38.076958 9494 raft_consensus.cc:1203] T 836882aa1fac4721be3cf096540eedc4 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:38.082648 10497 raft_consensus.cc:729] T 49b580d648014ce899fee8b3b329c47d P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:38.082895 10497 consensus_queue.cc:260] T 49b580d648014ce899fee8b3b329c47d P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:38.083252 10497 ts_tablet_manager.cc:1398] T 49b580d648014ce899fee8b3b329c47d P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20240503 01:59:38.083761 7034 tablet_copy_service.cc:342] P 63331f455be049c1bdd6c23c59ac1bef: Request end of tablet copy session b7062fbbc2b54fff9e99ec4ea5562c76-49b580d648014ce899fee8b3b329c47d received from {username='slave'} at 127.0.99.135:36345
I20240503 01:59:38.083845 7034 tablet_copy_service.cc:433] P 63331f455be049c1bdd6c23c59ac1bef: ending tablet copy session b7062fbbc2b54fff9e99ec4ea5562c76-49b580d648014ce899fee8b3b329c47d on tablet 49b580d648014ce899fee8b3b329c47d with peer b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 01:59:38.087584 10463 ts_tablet_manager.cc:902] T 6d4480c8de204f5397dbdcff91db97f6 P 12e8c683db054df3b75c6285b98a5ad8: Initiating tablet copy from peer 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:38.088148 10463 tablet_copy_client.cc:308] T 6d4480c8de204f5397dbdcff91db97f6 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36271
I20240503 01:59:38.088548 7295 tablet_copy_service.cc:140] P 1d0e738c34af40fa9950c6f086619b3a: Received BeginTabletCopySession request for tablet 6d4480c8de204f5397dbdcff91db97f6 from peer 12e8c683db054df3b75c6285b98a5ad8 ({username='slave'} at 127.0.99.133:37177)
I20240503 01:59:38.088644 7295 tablet_copy_service.cc:161] P 1d0e738c34af40fa9950c6f086619b3a: Beginning new tablet copy session on tablet 6d4480c8de204f5397dbdcff91db97f6 from peer 12e8c683db054df3b75c6285b98a5ad8 at {username='slave'} at 127.0.99.133:37177: session id = 12e8c683db054df3b75c6285b98a5ad8-6d4480c8de204f5397dbdcff91db97f6
I20240503 01:59:38.089537 7295 tablet_copy_source_session.cc:205] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.091002 10463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d4480c8de204f5397dbdcff91db97f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.093510 10463 tablet_copy_client.cc:783] T 6d4480c8de204f5397dbdcff91db97f6 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.093657 10463 tablet_copy_client.cc:647] T 6d4480c8de204f5397dbdcff91db97f6 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.096458 10463 tablet_copy_client.cc:523] T 6d4480c8de204f5397dbdcff91db97f6 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.097918 10463 tablet_bootstrap.cc:492] T 6d4480c8de204f5397dbdcff91db97f6 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:38.103823 10292 tablet_bootstrap.cc:492] T ccbd932e20b44dc5aceb86c86d7fda6b P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.109555 10365 tablet_bootstrap.cc:492] T 4600418bf64c4a0d962831c6924c46e1 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.109895 10365 tablet_bootstrap.cc:492] T 4600418bf64c4a0d962831c6924c46e1 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap complete.
I20240503 01:59:38.110162 10365 ts_tablet_manager.cc:1367] T 4600418bf64c4a0d962831c6924c46e1 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.068s user 0.021s sys 0.008s
I20240503 01:59:38.110692 10365 raft_consensus.cc:348] T 4600418bf64c4a0d962831c6924c46e1 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:38.110826 10365 raft_consensus.cc:729] T 4600418bf64c4a0d962831c6924c46e1 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:38.110993 10365 consensus_queue.cc:260] T 4600418bf64c4a0d962831c6924c46e1 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:38.111377 10365 ts_tablet_manager.cc:1398] T 4600418bf64c4a0d962831c6924c46e1 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:38.112649 7295 tablet_copy_service.cc:342] P 1d0e738c34af40fa9950c6f086619b3a: Request end of tablet copy session db1a0000aaa346f79c0aa4b6c5105191-4600418bf64c4a0d962831c6924c46e1 received from {username='slave'} at 127.0.99.136:38163
I20240503 01:59:38.112735 7295 tablet_copy_service.cc:433] P 1d0e738c34af40fa9950c6f086619b3a: ending tablet copy session db1a0000aaa346f79c0aa4b6c5105191-4600418bf64c4a0d962831c6924c46e1 on tablet 4600418bf64c4a0d962831c6924c46e1 with peer db1a0000aaa346f79c0aa4b6c5105191
I20240503 01:59:38.118851 10292 tablet_bootstrap.cc:492] T ccbd932e20b44dc5aceb86c86d7fda6b P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap complete.
I20240503 01:59:38.119093 10292 ts_tablet_manager.cc:1367] T ccbd932e20b44dc5aceb86c86d7fda6b P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.061s user 0.023s sys 0.004s
I20240503 01:59:38.119549 10292 raft_consensus.cc:348] T ccbd932e20b44dc5aceb86c86d7fda6b P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:38.119668 10292 raft_consensus.cc:729] T ccbd932e20b44dc5aceb86c86d7fda6b P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:38.119789 10292 consensus_queue.cc:260] T ccbd932e20b44dc5aceb86c86d7fda6b P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:38.120047 10292 ts_tablet_manager.cc:1398] T ccbd932e20b44dc5aceb86c86d7fda6b P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:38.123193 7165 tablet_copy_service.cc:342] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Request end of tablet copy session 12e8c683db054df3b75c6285b98a5ad8-ccbd932e20b44dc5aceb86c86d7fda6b received from {username='slave'} at 127.0.99.133:44539
I20240503 01:59:38.123292 7165 tablet_copy_service.cc:433] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: ending tablet copy session 12e8c683db054df3b75c6285b98a5ad8-ccbd932e20b44dc5aceb86c86d7fda6b on tablet ccbd932e20b44dc5aceb86c86d7fda6b with peer 12e8c683db054df3b75c6285b98a5ad8
I20240503 01:59:38.123277 6737 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:38.131726 6863 tablet_service.cc:1467] Processing CreateTablet for tablet e1f3ae5943604a868467244691d209f7 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=5108cdf748db49d3858f16e7dd630a09]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:38.132104 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1f3ae5943604a868467244691d209f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.132277 9736 tablet_service.cc:1467] Processing CreateTablet for tablet 4eccd41fc53241bb8496354405c92ea6 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=5108cdf748db49d3858f16e7dd630a09]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:38.132546 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eccd41fc53241bb8496354405c92ea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.132790 9474 tablet_service.cc:1467] Processing CreateTablet for tablet e1f3ae5943604a868467244691d209f7 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=5108cdf748db49d3858f16e7dd630a09]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:38.133042 9474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1f3ae5943604a868467244691d209f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.134622 10528 tablet_bootstrap.cc:492] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:38.134822 7255 tablet_service.cc:1467] Processing CreateTablet for tablet 9cc3435afe5144f9a52a286ac86d234f (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=5108cdf748db49d3858f16e7dd630a09]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:38.135105 7255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cc3435afe5144f9a52a286ac86d234f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.135502 9473 tablet_service.cc:1467] Processing CreateTablet for tablet 9cc3435afe5144f9a52a286ac86d234f (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=5108cdf748db49d3858f16e7dd630a09]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:38.135576 10528 tablet_bootstrap.cc:654] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:38.135712 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cc3435afe5144f9a52a286ac86d234f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.137403 10569 tablet_bootstrap.cc:492] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:38.137867 10527 tablet_bootstrap.cc:492] T e1f3ae5943604a868467244691d209f7 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:38.138571 10569 tablet_bootstrap.cc:654] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:38.138756 9625 raft_consensus.cc:1203] T 5cb5072e91534671ad3b4dd85f93fa5b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:38.138850 10527 tablet_bootstrap.cc:654] T e1f3ae5943604a868467244691d209f7 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:38.139497 9604 tablet_service.cc:1467] Processing CreateTablet for tablet 4eccd41fc53241bb8496354405c92ea6 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=5108cdf748db49d3858f16e7dd630a09]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:38.139742 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eccd41fc53241bb8496354405c92ea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.139765 9343 tablet_service.cc:1467] Processing CreateTablet for tablet 4eccd41fc53241bb8496354405c92ea6 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=5108cdf748db49d3858f16e7dd630a09]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:38.140025 9343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eccd41fc53241bb8496354405c92ea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.142081 10529 tablet_bootstrap.cc:492] T 4eccd41fc53241bb8496354405c92ea6 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:38.142483 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 9cc3435afe5144f9a52a286ac86d234f (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=5108cdf748db49d3858f16e7dd630a09]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:38.142771 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cc3435afe5144f9a52a286ac86d234f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.143093 10529 tablet_bootstrap.cc:654] T 4eccd41fc53241bb8496354405c92ea6 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:38.143353 10527 tablet_bootstrap.cc:492] T e1f3ae5943604a868467244691d209f7 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:38.143455 10527 ts_tablet_manager.cc:1367] T e1f3ae5943604a868467244691d209f7 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:38.143987 10527 raft_consensus.cc:348] T e1f3ae5943604a868467244691d209f7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:38.144126 10527 raft_consensus.cc:374] T e1f3ae5943604a868467244691d209f7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:38.144165 10527 raft_consensus.cc:729] T e1f3ae5943604a868467244691d209f7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:38.144294 10527 consensus_queue.cc:260] T e1f3ae5943604a868467244691d209f7 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:38.144639 10527 ts_tablet_manager.cc:1398] T e1f3ae5943604a868467244691d209f7 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:38.144834 10527 tablet_bootstrap.cc:492] T 9cc3435afe5144f9a52a286ac86d234f P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:38.145146 7124 tablet_service.cc:1467] Processing CreateTablet for tablet e1f3ae5943604a868467244691d209f7 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=5108cdf748db49d3858f16e7dd630a09]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:38.145467 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1f3ae5943604a868467244691d209f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.145815 10527 tablet_bootstrap.cc:654] T 9cc3435afe5144f9a52a286ac86d234f P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:38.146112 10463 tablet_bootstrap.cc:492] T 6d4480c8de204f5397dbdcff91db97f6 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.146386 10463 tablet_bootstrap.cc:492] T 6d4480c8de204f5397dbdcff91db97f6 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap complete.
I20240503 01:59:38.146646 10463 ts_tablet_manager.cc:1367] T 6d4480c8de204f5397dbdcff91db97f6 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.049s user 0.023s sys 0.004s
I20240503 01:59:38.147133 10463 raft_consensus.cc:348] T 6d4480c8de204f5397dbdcff91db97f6 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:38.147338 10463 raft_consensus.cc:729] T 6d4480c8de204f5397dbdcff91db97f6 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:38.147562 10463 consensus_queue.cc:260] T 6d4480c8de204f5397dbdcff91db97f6 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:38.147701 10527 tablet_bootstrap.cc:492] T 9cc3435afe5144f9a52a286ac86d234f P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:38.147786 10527 ts_tablet_manager.cc:1367] T 9cc3435afe5144f9a52a286ac86d234f P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:38.148262 10527 raft_consensus.cc:348] T 9cc3435afe5144f9a52a286ac86d234f P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:38.148367 10527 raft_consensus.cc:374] T 9cc3435afe5144f9a52a286ac86d234f P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:38.148409 10527 raft_consensus.cc:729] T 9cc3435afe5144f9a52a286ac86d234f P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:38.148532 10463 ts_tablet_manager.cc:1398] T 6d4480c8de204f5397dbdcff91db97f6 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:38.148550 10527 consensus_queue.cc:260] T 9cc3435afe5144f9a52a286ac86d234f P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:38.148833 10527 ts_tablet_manager.cc:1398] T 9cc3435afe5144f9a52a286ac86d234f P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:38.149267 7295 tablet_copy_service.cc:342] P 1d0e738c34af40fa9950c6f086619b3a: Request end of tablet copy session 12e8c683db054df3b75c6285b98a5ad8-6d4480c8de204f5397dbdcff91db97f6 received from {username='slave'} at 127.0.99.133:37177
I20240503 01:59:38.149351 7295 tablet_copy_service.cc:433] P 1d0e738c34af40fa9950c6f086619b3a: ending tablet copy session 12e8c683db054df3b75c6285b98a5ad8-6d4480c8de204f5397dbdcff91db97f6 on tablet 6d4480c8de204f5397dbdcff91db97f6 with peer 12e8c683db054df3b75c6285b98a5ad8
I20240503 01:59:38.150759 10528 tablet_bootstrap.cc:492] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:38.150887 10528 ts_tablet_manager.cc:1367] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 01:59:38.151443 10528 raft_consensus.cc:348] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:38.151502 10359 raft_consensus.cc:482] T 9cc3435afe5144f9a52a286ac86d234f P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:38.151574 10528 raft_consensus.cc:374] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:38.151613 10528 raft_consensus.cc:729] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:38.151593 10359 raft_consensus.cc:504] T 9cc3435afe5144f9a52a286ac86d234f P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:38.151757 10528 consensus_queue.cc:260] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:38.152055 10528 ts_tablet_manager.cc:1398] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:38.152071 10359 leader_election.cc:288] T 9cc3435afe5144f9a52a286ac86d234f P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:38.152748 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cc3435afe5144f9a52a286ac86d234f" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:38.153002 9362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cc3435afe5144f9a52a286ac86d234f" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
W20240503 01:59:38.153268 9428 leader_election.cc:341] T 9cc3435afe5144f9a52a286ac86d234f P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): Illegal state: must be running to vote when last-logged opid is not known
W20240503 01:59:38.153868 9429 leader_election.cc:341] T 9cc3435afe5144f9a52a286ac86d234f P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Illegal state: must be running to vote when last-logged opid is not known
I20240503 01:59:38.153949 9429 leader_election.cc:302] T 9cc3435afe5144f9a52a286ac86d234f P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a; no voters: 12e8c683db054df3b75c6285b98a5ad8, 1d0e738c34af40fa9950c6f086619b3a
I20240503 01:59:38.154091 10359 raft_consensus.cc:2726] T 9cc3435afe5144f9a52a286ac86d234f P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 01:59:38.154826 10569 tablet_bootstrap.cc:492] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:38.154937 10569 ts_tablet_manager.cc:1367] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.001s
I20240503 01:59:38.155332 10569 raft_consensus.cc:348] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:38.155436 10569 raft_consensus.cc:374] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:38.155457 10569 raft_consensus.cc:729] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:38.155565 10569 consensus_queue.cc:260] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:38.155829 10569 ts_tablet_manager.cc:1398] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:38.156252 10567 tablet_bootstrap.cc:492] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:38.157112 10567 tablet_bootstrap.cc:654] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:38.157729 10572 tablet_bootstrap.cc:492] T 4eccd41fc53241bb8496354405c92ea6 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:38.158212 10567 tablet_bootstrap.cc:492] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:38.158341 10567 ts_tablet_manager.cc:1367] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:38.159029 10572 tablet_bootstrap.cc:654] T 4eccd41fc53241bb8496354405c92ea6 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:38.159279 10567 raft_consensus.cc:348] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:38.159452 10567 raft_consensus.cc:374] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:38.159528 10567 raft_consensus.cc:729] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:38.159708 10567 consensus_queue.cc:260] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:38.159857 10529 tablet_bootstrap.cc:492] T 4eccd41fc53241bb8496354405c92ea6 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:38.159956 10529 ts_tablet_manager.cc:1367] T 4eccd41fc53241bb8496354405c92ea6 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240503 01:59:38.160488 10529 raft_consensus.cc:348] T 4eccd41fc53241bb8496354405c92ea6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:38.160614 10529 raft_consensus.cc:374] T 4eccd41fc53241bb8496354405c92ea6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:38.160645 10529 raft_consensus.cc:729] T 4eccd41fc53241bb8496354405c92ea6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:38.160787 10529 consensus_queue.cc:260] T 4eccd41fc53241bb8496354405c92ea6 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:38.161073 10529 ts_tablet_manager.cc:1398] T 4eccd41fc53241bb8496354405c92ea6 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:38.161332 10567 ts_tablet_manager.cc:1398] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:38.165856 10572 tablet_bootstrap.cc:492] T 4eccd41fc53241bb8496354405c92ea6 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:38.165967 10572 ts_tablet_manager.cc:1367] T 4eccd41fc53241bb8496354405c92ea6 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 01:59:38.166275 10575 tablet_bootstrap.cc:492] T e1f3ae5943604a868467244691d209f7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:38.166227 7274 consensus_queue.cc:237] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:38.166471 10572 raft_consensus.cc:348] T 4eccd41fc53241bb8496354405c92ea6 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:38.166637 10572 raft_consensus.cc:374] T 4eccd41fc53241bb8496354405c92ea6 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:38.166682 10572 raft_consensus.cc:729] T 4eccd41fc53241bb8496354405c92ea6 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:38.166827 10572 consensus_queue.cc:260] T 4eccd41fc53241bb8496354405c92ea6 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:38.167213 10572 ts_tablet_manager.cc:1398] T 4eccd41fc53241bb8496354405c92ea6 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:38.167428 10572 tablet_bootstrap.cc:492] T 9cc3435afe5144f9a52a286ac86d234f P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:38.168510 10572 tablet_bootstrap.cc:654] T 9cc3435afe5144f9a52a286ac86d234f P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
W20240503 01:59:38.168823 7211 consensus_peers.cc:485] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c029253a832e46a48ae16691fc044502. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.168920 6881 raft_consensus.cc:1261] T c029253a832e46a48ae16691fc044502 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:38.169116 7143 raft_consensus.cc:1261] T c029253a832e46a48ae16691fc044502 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:38.169284 10575 tablet_bootstrap.cc:654] T e1f3ae5943604a868467244691d209f7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:38.169612 10100 consensus_queue.cc:1035] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:38.169826 10389 consensus_queue.cc:1035] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:38.170164 10572 tablet_bootstrap.cc:492] T 9cc3435afe5144f9a52a286ac86d234f P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:38.170244 10572 ts_tablet_manager.cc:1367] T 9cc3435afe5144f9a52a286ac86d234f P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:38.171500 10575 tablet_bootstrap.cc:492] T e1f3ae5943604a868467244691d209f7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:38.171525 10572 raft_consensus.cc:348] T 9cc3435afe5144f9a52a286ac86d234f P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:38.171648 10572 raft_consensus.cc:374] T 9cc3435afe5144f9a52a286ac86d234f P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:38.171689 10572 raft_consensus.cc:729] T 9cc3435afe5144f9a52a286ac86d234f P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:38.171707 10575 ts_tablet_manager.cc:1367] T e1f3ae5943604a868467244691d209f7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240503 01:59:38.171804 10572 consensus_queue.cc:260] T 9cc3435afe5144f9a52a286ac86d234f P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:38.172106 10572 ts_tablet_manager.cc:1398] T 9cc3435afe5144f9a52a286ac86d234f P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:38.172286 10575 raft_consensus.cc:348] T e1f3ae5943604a868467244691d209f7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:38.172396 10575 raft_consensus.cc:374] T e1f3ae5943604a868467244691d209f7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:38.172430 10575 raft_consensus.cc:729] T e1f3ae5943604a868467244691d209f7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:38.172544 10575 consensus_queue.cc:260] T e1f3ae5943604a868467244691d209f7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:38.173203 10389 raft_consensus.cc:2932] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:38.173799 7143 raft_consensus.cc:2932] T c029253a832e46a48ae16691fc044502 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:38.174373 10575 ts_tablet_manager.cc:1398] T e1f3ae5943604a868467244691d209f7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 01:59:38.176048 6881 raft_consensus.cc:2932] T c029253a832e46a48ae16691fc044502 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:38.177983 6737 catalog_manager.cc:5484] T c029253a832e46a48ae16691fc044502 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:38.200634 7274 consensus_queue.cc:237] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:38.202059 6881 raft_consensus.cc:1261] T 46f4a187af854867bcd509c17ced5cd7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:38.202103 7012 raft_consensus.cc:1261] T 46f4a187af854867bcd509c17ced5cd7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:38.202268 7210 consensus_peers.cc:485] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a -> Peer db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): Couldn't send request to peer db1a0000aaa346f79c0aa4b6c5105191. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 46f4a187af854867bcd509c17ced5cd7. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.202430 10100 consensus_queue.cc:1035] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:38.202618 10585 consensus_queue.cc:1035] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:38.205375 10585 raft_consensus.cc:2932] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:38.206879 7012 raft_consensus.cc:2932] T 46f4a187af854867bcd509c17ced5cd7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:38.210175 6738 catalog_manager.cc:5484] T 46f4a187af854867bcd509c17ced5cd7 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:38.211076 6881 raft_consensus.cc:2932] T 46f4a187af854867bcd509c17ced5cd7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:38.218163 9757 raft_consensus.cc:1203] T c6833c997d284063955d0b9b09ccb3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:38.239032 7143 consensus_queue.cc:237] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:38.240438 6882 raft_consensus.cc:1261] T 8295ea038a88485095544cc608b000c2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:38.240916 9092 consensus_queue.cc:1035] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:38.241274 7012 raft_consensus.cc:1261] T 8295ea038a88485095544cc608b000c2 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:38.241672 9092 consensus_queue.cc:1035] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:38.243932 10620 raft_consensus.cc:2932] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
W20240503 01:59:38.244624 7080 consensus_peers.cc:485] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8295ea038a88485095544cc608b000c2. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.246037 6882 raft_consensus.cc:2932] T 8295ea038a88485095544cc608b000c2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:38.246254 7012 raft_consensus.cc:2932] T 8295ea038a88485095544cc608b000c2 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:38.247198 6740 catalog_manager.cc:5484] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:38.266176 9970 raft_consensus.cc:482] T 4eccd41fc53241bb8496354405c92ea6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:38.266388 9970 raft_consensus.cc:504] T 4eccd41fc53241bb8496354405c92ea6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:38.267079 9970 leader_election.cc:288] T 4eccd41fc53241bb8496354405c92ea6 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:38.267542 9362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eccd41fc53241bb8496354405c92ea6" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:38.267771 9362 raft_consensus.cc:2454] T 4eccd41fc53241bb8496354405c92ea6 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 01:59:38.267689 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eccd41fc53241bb8496354405c92ea6" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:38.267921 9757 raft_consensus.cc:2454] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 01:59:38.268110 9561 leader_election.cc:302] T 4eccd41fc53241bb8496354405c92ea6 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:38.268302 9970 raft_consensus.cc:2781] T 4eccd41fc53241bb8496354405c92ea6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:38.268353 9970 raft_consensus.cc:482] T 4eccd41fc53241bb8496354405c92ea6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:38.268385 9970 raft_consensus.cc:3037] T 4eccd41fc53241bb8496354405c92ea6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:38.269977 9970 raft_consensus.cc:504] T 4eccd41fc53241bb8496354405c92ea6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:38.270854 9970 leader_election.cc:288] T 4eccd41fc53241bb8496354405c92ea6 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Requested vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:38.271216 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eccd41fc53241bb8496354405c92ea6" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:38.271225 9362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eccd41fc53241bb8496354405c92ea6" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:38.271346 9757 raft_consensus.cc:3037] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:38.271369 9362 raft_consensus.cc:3037] T 4eccd41fc53241bb8496354405c92ea6 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:38.272598 9362 raft_consensus.cc:2454] T 4eccd41fc53241bb8496354405c92ea6 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 01:59:38.272598 9757 raft_consensus.cc:2454] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 01:59:38.273077 9560 leader_election.cc:302] T 4eccd41fc53241bb8496354405c92ea6 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:38.273486 9970 raft_consensus.cc:2781] T 4eccd41fc53241bb8496354405c92ea6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:38.273598 9970 raft_consensus.cc:686] T 4eccd41fc53241bb8496354405c92ea6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 LEADER]: Becoming Leader. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Running, Role: LEADER
I20240503 01:59:38.273766 9970 consensus_queue.cc:237] T 4eccd41fc53241bb8496354405c92ea6 P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:38.274775 7143 consensus_queue.cc:237] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:38.275760 6739 catalog_manager.cc:5484] T 4eccd41fc53241bb8496354405c92ea6 P b7062fbbc2b54fff9e99ec4ea5562c76 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:38.276458 6882 raft_consensus.cc:1261] T db656aaa7b104b3182350de320f67839 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:38.276783 9092 consensus_queue.cc:1035] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:38.277657 7012 raft_consensus.cc:1261] T db656aaa7b104b3182350de320f67839 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:38.278725 9092 consensus_queue.cc:1035] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:38.278913 7078 consensus_peers.cc:485] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db656aaa7b104b3182350de320f67839. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:38.280527 10298 raft_consensus.cc:2932] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:38.280987 7012 raft_consensus.cc:2932] T db656aaa7b104b3182350de320f67839 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:38.284271 6739 catalog_manager.cc:5484] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:38.287046 6882 raft_consensus.cc:2932] T db656aaa7b104b3182350de320f67839 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:38.304731 9362 raft_consensus.cc:1203] T a5e23eebc629494cbb8262bc123a720a P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:38.307442 6738 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240503 01:59:38.307601 6738 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240503 01:59:38.309978 6738 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 98be5e3f127842d1b6380f13cb6842dd
I20240503 01:59:38.310259 6738 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 4 replicas of tablet 46f4a187af854867bcd509c17ced5cd7
I20240503 01:59:38.310480 6738 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 4 replicas of tablet 836882aa1fac4721be3cf096540eedc4
I20240503 01:59:38.310804 9736 tablet_service.cc:1514] Processing DeleteTablet for tablet 46f4a187af854867bcd509c17ced5cd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:38 UTC) from {username='slave'} at 127.0.0.1:48152
I20240503 01:59:38.311365 6992 tablet_service.cc:1514] Processing DeleteTablet for tablet 98be5e3f127842d1b6380f13cb6842dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:38 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:38.311462 6994 tablet_service.cc:1514] Processing DeleteTablet for tablet 46f4a187af854867bcd509c17ced5cd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:38 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:38.311583 10471 tablet_replica.cc:330] T 98be5e3f127842d1b6380f13cb6842dd P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:38.311570 9473 tablet_service.cc:1514] Processing DeleteTablet for tablet 836882aa1fac4721be3cf096540eedc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:38 UTC) from {username='slave'} at 127.0.0.1:54678
I20240503 01:59:38.311734 10496 tablet_replica.cc:330] T 836882aa1fac4721be3cf096540eedc4 P bb5364d730cb4489a0bfc6c270b7450a: stopping tablet replica
I20240503 01:59:38.311728 10471 raft_consensus.cc:2229] T 98be5e3f127842d1b6380f13cb6842dd P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.311852 10471 raft_consensus.cc:2258] T 98be5e3f127842d1b6380f13cb6842dd P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.311846 10496 raft_consensus.cc:2229] T 836882aa1fac4721be3cf096540eedc4 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.311939 10496 raft_consensus.cc:2258] T 836882aa1fac4721be3cf096540eedc4 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.312278 10471 ts_tablet_manager.cc:1875] T 98be5e3f127842d1b6380f13cb6842dd P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:38.312582 10496 ts_tablet_manager.cc:1875] T 836882aa1fac4721be3cf096540eedc4 P bb5364d730cb4489a0bfc6c270b7450a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:38.311597 6993 tablet_service.cc:1514] Processing DeleteTablet for tablet 836882aa1fac4721be3cf096540eedc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:38 UTC) from {username='slave'} at 127.0.0.1:35240
W20240503 01:59:38.314272 6726 catalog_manager.cc:4797] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): delete failed for tablet 46f4a187af854867bcd509c17ced5cd7 because the tablet was not found. No further retry: Not found: Tablet not found: 46f4a187af854867bcd509c17ced5cd7
I20240503 01:59:38.314551 10471 ts_tablet_manager.cc:1888] T 98be5e3f127842d1b6380f13cb6842dd P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:38.314622 10471 log.cc:1196] T 98be5e3f127842d1b6380f13cb6842dd P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/98be5e3f127842d1b6380f13cb6842dd
I20240503 01:59:38.314672 10496 ts_tablet_manager.cc:1888] T 836882aa1fac4721be3cf096540eedc4 P bb5364d730cb4489a0bfc6c270b7450a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:38.314738 10496 log.cc:1196] T 836882aa1fac4721be3cf096540eedc4 P bb5364d730cb4489a0bfc6c270b7450a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/836882aa1fac4721be3cf096540eedc4
I20240503 01:59:38.314903 10471 ts_tablet_manager.cc:1909] T 98be5e3f127842d1b6380f13cb6842dd P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:38.314939 7255 tablet_service.cc:1514] Processing DeleteTablet for tablet 46f4a187af854867bcd509c17ced5cd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:38 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:38.315029 10496 ts_tablet_manager.cc:1909] T 836882aa1fac4721be3cf096540eedc4 P bb5364d730cb4489a0bfc6c270b7450a: Deleting consensus metadata
I20240503 01:59:38.314953 7254 tablet_service.cc:1514] Processing DeleteTablet for tablet 836882aa1fac4721be3cf096540eedc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:38 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:38.315169 10468 tablet_replica.cc:330] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:38.315301 10468 raft_consensus.cc:2229] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:38.315553 10468 raft_consensus.cc:2258] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.315928 7124 tablet_service.cc:1514] Processing DeleteTablet for tablet 98be5e3f127842d1b6380f13cb6842dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:38 UTC) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:38.316093 7125 tablet_service.cc:1514] Processing DeleteTablet for tablet 836882aa1fac4721be3cf096540eedc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:38 UTC) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:38.316090 6725 catalog_manager.cc:4830] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): tablet 836882aa1fac4721be3cf096540eedc4 (table rebalance_test_table_8 [id=0f89521df2e14bcd8bbc6f1041196932]) successfully deleted
I20240503 01:59:38.316241 10470 tablet_replica.cc:330] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:38.316290 10468 ts_tablet_manager.cc:1875] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:38.316380 10470 raft_consensus.cc:2229] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:38.316569 10470 raft_consensus.cc:2258] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.316838 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet 98be5e3f127842d1b6380f13cb6842dd (table rebalance_test_table_8 [id=0f89521df2e14bcd8bbc6f1041196932]) successfully deleted
I20240503 01:59:38.317050 10471 tablet_replica.cc:330] T 46f4a187af854867bcd509c17ced5cd7 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:38.317101 10470 ts_tablet_manager.cc:1875] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:38.317170 10471 raft_consensus.cc:2229] T 46f4a187af854867bcd509c17ced5cd7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.317548 10471 raft_consensus.cc:2258] T 46f4a187af854867bcd509c17ced5cd7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.318619 10471 ts_tablet_manager.cc:1875] T 46f4a187af854867bcd509c17ced5cd7 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:38.319257 10470 ts_tablet_manager.cc:1888] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:38.319272 10468 ts_tablet_manager.cc:1888] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:38.319332 10470 log.cc:1196] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/98be5e3f127842d1b6380f13cb6842dd
I20240503 01:59:38.319335 10468 log.cc:1196] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/46f4a187af854867bcd509c17ced5cd7
I20240503 01:59:38.319628 10470 ts_tablet_manager.cc:1909] T 98be5e3f127842d1b6380f13cb6842dd P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting consensus metadata
I20240503 01:59:38.319629 10468 ts_tablet_manager.cc:1909] T 46f4a187af854867bcd509c17ced5cd7 P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:38.320362 6863 tablet_service.cc:1514] Processing DeleteTablet for tablet 98be5e3f127842d1b6380f13cb6842dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:38 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:38.320560 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet 46f4a187af854867bcd509c17ced5cd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:38 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:38.320645 9955 tablet_replica.cc:330] T 98be5e3f127842d1b6380f13cb6842dd P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:38.320866 9955 raft_consensus.cc:2229] T 98be5e3f127842d1b6380f13cb6842dd P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.321046 9955 raft_consensus.cc:2258] T 98be5e3f127842d1b6380f13cb6842dd P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.321128 10470 tablet_replica.cc:330] T 836882aa1fac4721be3cf096540eedc4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:38.321352 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet 98be5e3f127842d1b6380f13cb6842dd (table rebalance_test_table_8 [id=0f89521df2e14bcd8bbc6f1041196932]) successfully deleted
I20240503 01:59:38.321247 10470 raft_consensus.cc:2229] T 836882aa1fac4721be3cf096540eedc4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.321600 10470 raft_consensus.cc:2258] T 836882aa1fac4721be3cf096540eedc4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.321640 9955 ts_tablet_manager.cc:1875] T 98be5e3f127842d1b6380f13cb6842dd P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:38.321965 10470 ts_tablet_manager.cc:1875] T 836882aa1fac4721be3cf096540eedc4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:38.325695 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet 46f4a187af854867bcd509c17ced5cd7 (table rebalance_test_table_8 [id=0f89521df2e14bcd8bbc6f1041196932]) successfully deleted
I20240503 01:59:38.327029 9955 ts_tablet_manager.cc:1888] T 98be5e3f127842d1b6380f13cb6842dd P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:38.327198 9955 log.cc:1196] T 98be5e3f127842d1b6380f13cb6842dd P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/98be5e3f127842d1b6380f13cb6842dd
I20240503 01:59:38.327549 9955 ts_tablet_manager.cc:1909] T 98be5e3f127842d1b6380f13cb6842dd P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:38.328516 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 98be5e3f127842d1b6380f13cb6842dd (table rebalance_test_table_8 [id=0f89521df2e14bcd8bbc6f1041196932]) successfully deleted
I20240503 01:59:38.328856 9955 tablet_replica.cc:330] T 46f4a187af854867bcd509c17ced5cd7 P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:38.329056 9955 raft_consensus.cc:2229] T 46f4a187af854867bcd509c17ced5cd7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.329361 9955 raft_consensus.cc:2258] T 46f4a187af854867bcd509c17ced5cd7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.330507 10468 tablet_replica.cc:330] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:38.330940 9955 ts_tablet_manager.cc:1875] T 46f4a187af854867bcd509c17ced5cd7 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:38.331439 9362 raft_consensus.cc:1203] T c35ecf8890b64caaac7e5ea9f6e8f989 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:38.332737 10468 raft_consensus.cc:2229] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:38.333108 10468 raft_consensus.cc:2258] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.333259 9955 ts_tablet_manager.cc:1888] T 46f4a187af854867bcd509c17ced5cd7 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:38.333338 9955 log.cc:1196] T 46f4a187af854867bcd509c17ced5cd7 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/46f4a187af854867bcd509c17ced5cd7
I20240503 01:59:38.333598 9955 ts_tablet_manager.cc:1909] T 46f4a187af854867bcd509c17ced5cd7 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:38.333928 10468 ts_tablet_manager.cc:1875] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:38.334297 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 46f4a187af854867bcd509c17ced5cd7 (table rebalance_test_table_8 [id=0f89521df2e14bcd8bbc6f1041196932]) successfully deleted
I20240503 01:59:38.335568 10470 ts_tablet_manager.cc:1888] T 836882aa1fac4721be3cf096540eedc4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:38.335704 10470 log.cc:1196] T 836882aa1fac4721be3cf096540eedc4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/836882aa1fac4721be3cf096540eedc4
I20240503 01:59:38.336136 10470 ts_tablet_manager.cc:1909] T 836882aa1fac4721be3cf096540eedc4 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting consensus metadata
I20240503 01:59:38.336629 10468 ts_tablet_manager.cc:1888] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:38.336707 10468 log.cc:1196] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/836882aa1fac4721be3cf096540eedc4
I20240503 01:59:38.337004 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet 836882aa1fac4721be3cf096540eedc4 (table rebalance_test_table_8 [id=0f89521df2e14bcd8bbc6f1041196932]) successfully deleted
I20240503 01:59:38.337072 10468 ts_tablet_manager.cc:1909] T 836882aa1fac4721be3cf096540eedc4 P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:38.338258 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet 836882aa1fac4721be3cf096540eedc4 (table rebalance_test_table_8 [id=0f89521df2e14bcd8bbc6f1041196932]) successfully deleted
I20240503 01:59:38.340373 10471 ts_tablet_manager.cc:1888] T 46f4a187af854867bcd509c17ced5cd7 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:38.340526 10471 log.cc:1196] T 46f4a187af854867bcd509c17ced5cd7 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/46f4a187af854867bcd509c17ced5cd7
I20240503 01:59:38.340983 10471 ts_tablet_manager.cc:1909] T 46f4a187af854867bcd509c17ced5cd7 P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:38.341645 10471 tablet_replica.cc:330] T 836882aa1fac4721be3cf096540eedc4 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:38.341754 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet 46f4a187af854867bcd509c17ced5cd7 (table rebalance_test_table_8 [id=0f89521df2e14bcd8bbc6f1041196932]) successfully deleted
I20240503 01:59:38.342659 10471 raft_consensus.cc:2229] T 836882aa1fac4721be3cf096540eedc4 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.342871 10471 raft_consensus.cc:2258] T 836882aa1fac4721be3cf096540eedc4 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.343436 10471 ts_tablet_manager.cc:1875] T 836882aa1fac4721be3cf096540eedc4 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:38.348402 10471 ts_tablet_manager.cc:1888] T 836882aa1fac4721be3cf096540eedc4 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:38.348567 10471 log.cc:1196] T 836882aa1fac4721be3cf096540eedc4 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/836882aa1fac4721be3cf096540eedc4
I20240503 01:59:38.349674 10545 raft_consensus.cc:482] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:38.349826 10545 raft_consensus.cc:504] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:38.349872 10471 ts_tablet_manager.cc:1909] T 836882aa1fac4721be3cf096540eedc4 P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:38.350667 10545 leader_election.cc:288] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:38.351228 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de80a938a89f471b8b9c0c4b1b4be326" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:38.351363 9494 raft_consensus.cc:2454] T de80a938a89f471b8b9c0c4b1b4be326 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 01:59:38.351572 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet 836882aa1fac4721be3cf096540eedc4 (table rebalance_test_table_8 [id=0f89521df2e14bcd8bbc6f1041196932]) successfully deleted
I20240503 01:59:38.352020 9690 leader_election.cc:302] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:38.352363 10545 raft_consensus.cc:2781] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:38.352411 10545 raft_consensus.cc:482] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:38.352445 10545 raft_consensus.cc:3037] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:38.353451 10545 raft_consensus.cc:504] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:38.353780 10545 leader_election.cc:288] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Requested vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:38.355118 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de80a938a89f471b8b9c0c4b1b4be326" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:38.355372 9494 raft_consensus.cc:3037] T de80a938a89f471b8b9c0c4b1b4be326 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:38.356307 9494 raft_consensus.cc:2454] T de80a938a89f471b8b9c0c4b1b4be326 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:38.356303 9626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de80a938a89f471b8b9c0c4b1b4be326" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:38.356444 9626 raft_consensus.cc:3037] T de80a938a89f471b8b9c0c4b1b4be326 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:38.356737 9690 leader_election.cc:302] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:38.356899 10545 raft_consensus.cc:2781] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:38.356969 10545 raft_consensus.cc:686] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Becoming Leader. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Running, Role: LEADER
I20240503 01:59:38.357107 10545 consensus_queue.cc:237] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:38.357650 9757 raft_consensus.cc:1203] T dec7e3036d384f5db51f50ba93acbbc6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:38.357750 9626 raft_consensus.cc:2454] T de80a938a89f471b8b9c0c4b1b4be326 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:38.356303 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de80a938a89f471b8b9c0c4b1b4be326" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:38.358486 9625 raft_consensus.cc:2362] T de80a938a89f471b8b9c0c4b1b4be326 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1. Re-sending same reply.
I20240503 01:59:38.358790 6738 catalog_manager.cc:5484] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: term changed from 0 to 1, leader changed from <none> to db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:38.361519 10273 raft_consensus.cc:482] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:38.361677 10273 raft_consensus.cc:504] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:38.362318 10273 leader_election.cc:288] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:38.363041 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f3ae5943604a868467244691d209f7" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:38.363210 7143 raft_consensus.cc:2454] T e1f3ae5943604a868467244691d209f7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:38.363610 6818 leader_election.cc:302] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:38.364668 10273 raft_consensus.cc:2781] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:38.364744 10273 raft_consensus.cc:482] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:38.364792 10273 raft_consensus.cc:3037] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:38.366073 10273 raft_consensus.cc:504] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:38.366437 10273 leader_election.cc:288] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Requested vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:38.366945 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f3ae5943604a868467244691d209f7" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:38.367098 7143 raft_consensus.cc:3037] T e1f3ae5943604a868467244691d209f7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:38.367106 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f3ae5943604a868467244691d209f7" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:38.367290 9493 raft_consensus.cc:3037] T e1f3ae5943604a868467244691d209f7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:38.368373 7143 raft_consensus.cc:2454] T e1f3ae5943604a868467244691d209f7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:38.368403 9493 raft_consensus.cc:2454] T e1f3ae5943604a868467244691d209f7 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:38.368705 6818 leader_election.cc:302] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:38.368906 10273 raft_consensus.cc:2781] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:38.368937 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f3ae5943604a868467244691d209f7" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:38.369064 9494 raft_consensus.cc:2362] T e1f3ae5943604a868467244691d209f7 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1. Re-sending same reply.
I20240503 01:59:38.369148 10273 raft_consensus.cc:686] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 01:59:38.369357 10273 consensus_queue.cc:237] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:38.371863 6738 catalog_manager.cc:5484] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:38.428351 10580 raft_consensus.cc:482] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:38.428563 10580 raft_consensus.cc:504] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:38.429174 10580 leader_election.cc:288] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:38.429695 9362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cc3435afe5144f9a52a286ac86d234f" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:38.429755 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cc3435afe5144f9a52a286ac86d234f" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:38.429880 9362 raft_consensus.cc:2454] T 9cc3435afe5144f9a52a286ac86d234f P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:38.429899 9494 raft_consensus.cc:2454] T 9cc3435afe5144f9a52a286ac86d234f P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:38.430289 7209 leader_election.cc:302] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:38.430512 10580 raft_consensus.cc:2781] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:38.430613 10580 raft_consensus.cc:482] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:38.430667 10580 raft_consensus.cc:3037] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:38.431746 10580 raft_consensus.cc:504] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:38.431918 9494 raft_consensus.cc:1203] T 8ed2793842464443826f83f60fb6dcd1 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:38.432157 10580 leader_election.cc:288] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Requested vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:38.432638 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cc3435afe5144f9a52a286ac86d234f" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:38.432756 9494 raft_consensus.cc:3037] T 9cc3435afe5144f9a52a286ac86d234f P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:38.433147 9362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cc3435afe5144f9a52a286ac86d234f" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:38.433295 9362 raft_consensus.cc:3037] T 9cc3435afe5144f9a52a286ac86d234f P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:38.433776 9494 raft_consensus.cc:2454] T 9cc3435afe5144f9a52a286ac86d234f P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:38.434361 9362 raft_consensus.cc:2454] T 9cc3435afe5144f9a52a286ac86d234f P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:38.434721 7209 leader_election.cc:302] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:38.434904 10580 raft_consensus.cc:2781] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:38.435007 10580 raft_consensus.cc:686] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 01:59:38.435173 10580 consensus_queue.cc:237] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:38.437039 6732 catalog_manager.cc:5484] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:38.455655 9625 raft_consensus.cc:1203] T 49b580d648014ce899fee8b3b329c47d P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:38.467233 9604 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:38.468941 7254 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:38.474732 9473 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:38.474732 6732 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:38.475778 6993 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 01:59:38.482993 9494 raft_consensus.cc:1261] T de80a938a89f471b8b9c0c4b1b4be326 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:38.483448 10547 consensus_queue.cc:1035] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:38.483783 9625 raft_consensus.cc:1261] T de80a938a89f471b8b9c0c4b1b4be326 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:38.484177 10547 consensus_queue.cc:1035] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:38.484596 7125 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:38.487814 9736 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:38.487825 6732 catalog_manager.cc:5576] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087) does not have the latest schema for tablet de80a938a89f471b8b9c0c4b1b4be326 (table rebalancer_dynamic_table [id=70b7076aa6314855b03666d69272e9fd]). Expected version 1 got 0
I20240503 01:59:38.488579 6735 catalog_manager.cc:5576] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471) does not have the latest schema for tablet de80a938a89f471b8b9c0c4b1b4be326 (table rebalancer_dynamic_table [id=70b7076aa6314855b03666d69272e9fd]). Expected version 1 got 0
I20240503 01:59:38.489369 10707 tablet.cc:1722] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:38.489780 10707 tablet.cc:1620] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:38.490397 10704 mvcc.cc:205] Tried to move back new op lower bound from 7023417665457602560 to 7023417664952123392. Current Snapshot: MvccSnapshot[applied={T|T < 7023417665457602560}]
I20240503 01:59:38.491948 6735 catalog_manager.cc:5576] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471) does not have the latest schema for tablet de80a938a89f471b8b9c0c4b1b4be326 (table rebalancer_dynamic_table [id=70b7076aa6314855b03666d69272e9fd]). Expected version 1 got 0
I20240503 01:59:38.498247 10492 mvcc.cc:205] Tried to move back new op lower bound from 7023417665457602560 to 7023417664952123392. Current Snapshot: MvccSnapshot[applied={T|T < 7023417665457602560}]
I20240503 01:59:38.499933 6732 catalog_manager.cc:5576] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087) does not have the latest schema for tablet de80a938a89f471b8b9c0c4b1b4be326 (table rebalancer_dynamic_table [id=70b7076aa6314855b03666d69272e9fd]). Expected version 1 got 0
I20240503 01:59:38.503628 6861 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:38.507539 9342 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:38.519075 6732 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=70b7076aa6314855b03666d69272e9fd] alter complete (version 1)
I20240503 01:59:38.527755 10708 tablet.cc:1722] T de80a938a89f471b8b9c0c4b1b4be326 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:38.530541 10708 tablet.cc:1620] T de80a938a89f471b8b9c0c4b1b4be326 P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:38.539455 10709 tablet.cc:1722] T de80a938a89f471b8b9c0c4b1b4be326 P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:38.540483 10709 tablet.cc:1620] T de80a938a89f471b8b9c0c4b1b4be326 P b7062fbbc2b54fff9e99ec4ea5562c76: MemRowSet was empty: no flush needed.
I20240503 01:59:38.566838 10710 ts_tablet_manager.cc:902] T 31b077072738410f8d7c139ad1b5aa9c P 5dcc33cc621a4cbbae155f3230fb7c46: Initiating tablet copy from peer db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:38.567692 10710 tablet_copy_client.cc:308] T 31b077072738410f8d7c139ad1b5aa9c P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:42053
I20240503 01:59:38.568533 9777 tablet_copy_service.cc:140] P db1a0000aaa346f79c0aa4b6c5105191: Received BeginTabletCopySession request for tablet 31b077072738410f8d7c139ad1b5aa9c from peer 5dcc33cc621a4cbbae155f3230fb7c46 ({username='slave'} at 127.0.99.129:48571)
I20240503 01:59:38.568672 9777 tablet_copy_service.cc:161] P db1a0000aaa346f79c0aa4b6c5105191: Beginning new tablet copy session on tablet 31b077072738410f8d7c139ad1b5aa9c from peer 5dcc33cc621a4cbbae155f3230fb7c46 at {username='slave'} at 127.0.99.129:48571: session id = 5dcc33cc621a4cbbae155f3230fb7c46-31b077072738410f8d7c139ad1b5aa9c
I20240503 01:59:38.569797 9777 tablet_copy_source_session.cc:205] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.571475 10710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31b077072738410f8d7c139ad1b5aa9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.571786 7274 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "46f4a187af854867bcd509c17ced5cd7"
dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51574
I20240503 01:59:38.574337 10710 tablet_copy_client.cc:783] T 31b077072738410f8d7c139ad1b5aa9c P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.574570 10710 tablet_copy_client.cc:647] T 31b077072738410f8d7c139ad1b5aa9c P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.576376 10710 tablet_copy_client.cc:523] T 31b077072738410f8d7c139ad1b5aa9c P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.578219 10710 tablet_bootstrap.cc:492] T 31b077072738410f8d7c139ad1b5aa9c P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:38.582756 10710 tablet_bootstrap.cc:492] T 31b077072738410f8d7c139ad1b5aa9c P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.583062 10710 tablet_bootstrap.cc:492] T 31b077072738410f8d7c139ad1b5aa9c P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap complete.
I20240503 01:59:38.583240 10710 ts_tablet_manager.cc:1367] T 31b077072738410f8d7c139ad1b5aa9c P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20240503 01:59:38.583724 10710 raft_consensus.cc:348] T 31b077072738410f8d7c139ad1b5aa9c P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:38.583868 10710 raft_consensus.cc:729] T 31b077072738410f8d7c139ad1b5aa9c P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:38.584044 10710 consensus_queue.cc:260] T 31b077072738410f8d7c139ad1b5aa9c P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:38.585489 10710 ts_tablet_manager.cc:1398] T 31b077072738410f8d7c139ad1b5aa9c P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:38.588023 9777 tablet_copy_service.cc:342] P db1a0000aaa346f79c0aa4b6c5105191: Request end of tablet copy session 5dcc33cc621a4cbbae155f3230fb7c46-31b077072738410f8d7c139ad1b5aa9c received from {username='slave'} at 127.0.99.129:48571
I20240503 01:59:38.588146 9777 tablet_copy_service.cc:433] P db1a0000aaa346f79c0aa4b6c5105191: ending tablet copy session 5dcc33cc621a4cbbae155f3230fb7c46-31b077072738410f8d7c139ad1b5aa9c on tablet 31b077072738410f8d7c139ad1b5aa9c with peer 5dcc33cc621a4cbbae155f3230fb7c46
I20240503 01:59:38.628347 6739 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:38.635537 10707 tablet.cc:1722] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:38.635898 10707 tablet.cc:1620] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:38.637969 10708 tablet.cc:1722] T de80a938a89f471b8b9c0c4b1b4be326 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:38.638326 10708 tablet.cc:1620] T de80a938a89f471b8b9c0c4b1b4be326 P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:38.641010 10709 tablet.cc:1722] T de80a938a89f471b8b9c0c4b1b4be326 P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:38.641355 10709 tablet.cc:1620] T de80a938a89f471b8b9c0c4b1b4be326 P b7062fbbc2b54fff9e99ec4ea5562c76: MemRowSet was empty: no flush needed.
I20240503 01:59:38.649660 9757 raft_consensus.cc:1203] T 4600418bf64c4a0d962831c6924c46e1 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:38.650211 6734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=70b7076aa6314855b03666d69272e9fd] alter complete (version 2)
I20240503 01:59:38.664335 9362 raft_consensus.cc:1203] T ccbd932e20b44dc5aceb86c86d7fda6b P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:38.668486 6993 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 01:59:38.670194 6861 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:38.676358 9362 raft_consensus.cc:1203] T 6d4480c8de204f5397dbdcff91db97f6 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:38.678473 7254 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:38.679611 9604 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:38.680352 9736 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:38.680526 9342 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:38.697641 9473 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:38.698702 7125 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:38.706914 9970 consensus_queue.cc:1035] T 4eccd41fc53241bb8496354405c92ea6 P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:38.724243 10768 ts_tablet_manager.cc:902] T c029253a832e46a48ae16691fc044502 P b7062fbbc2b54fff9e99ec4ea5562c76: Initiating tablet copy from peer 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:38.727280 10377 consensus_queue.cc:1035] T 4eccd41fc53241bb8496354405c92ea6 P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:38.727973 10768 tablet_copy_client.cc:308] T c029253a832e46a48ae16691fc044502 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36271
I20240503 01:59:38.730755 7295 tablet_copy_service.cc:140] P 1d0e738c34af40fa9950c6f086619b3a: Received BeginTabletCopySession request for tablet c029253a832e46a48ae16691fc044502 from peer b7062fbbc2b54fff9e99ec4ea5562c76 ({username='slave'} at 127.0.99.135:51921)
I20240503 01:59:38.730876 7295 tablet_copy_service.cc:161] P 1d0e738c34af40fa9950c6f086619b3a: Beginning new tablet copy session on tablet c029253a832e46a48ae16691fc044502 from peer b7062fbbc2b54fff9e99ec4ea5562c76 at {username='slave'} at 127.0.99.135:51921: session id = b7062fbbc2b54fff9e99ec4ea5562c76-c029253a832e46a48ae16691fc044502
I20240503 01:59:38.731771 10775 ts_tablet_manager.cc:902] T 8295ea038a88485095544cc608b000c2 P b7062fbbc2b54fff9e99ec4ea5562c76: Initiating tablet copy from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:38.731941 7295 tablet_copy_source_session.cc:205] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.733100 10768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c029253a832e46a48ae16691fc044502. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.733175 10775 tablet_copy_client.cc:308] T 8295ea038a88485095544cc608b000c2 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41979
I20240503 01:59:38.737794 7165 tablet_copy_service.cc:140] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Received BeginTabletCopySession request for tablet 8295ea038a88485095544cc608b000c2 from peer b7062fbbc2b54fff9e99ec4ea5562c76 ({username='slave'} at 127.0.99.135:42797)
I20240503 01:59:38.737931 7165 tablet_copy_service.cc:161] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Beginning new tablet copy session on tablet 8295ea038a88485095544cc608b000c2 from peer b7062fbbc2b54fff9e99ec4ea5562c76 at {username='slave'} at 127.0.99.135:42797: session id = b7062fbbc2b54fff9e99ec4ea5562c76-8295ea038a88485095544cc608b000c2
I20240503 01:59:38.738943 7165 tablet_copy_source_session.cc:205] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.739912 10775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8295ea038a88485095544cc608b000c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.752488 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:38.756889 10768 tablet_copy_client.cc:783] T c029253a832e46a48ae16691fc044502 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.757077 10768 tablet_copy_client.cc:647] T c029253a832e46a48ae16691fc044502 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.758435 10768 tablet_copy_client.cc:523] T c029253a832e46a48ae16691fc044502 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.759030 10775 tablet_copy_client.cc:783] T 8295ea038a88485095544cc608b000c2 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.759171 10775 tablet_copy_client.cc:647] T 8295ea038a88485095544cc608b000c2 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.759778 10768 tablet_bootstrap.cc:492] T c029253a832e46a48ae16691fc044502 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:38.760248 10707 tablet.cc:1722] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:38.761121 10709 tablet.cc:1722] T de80a938a89f471b8b9c0c4b1b4be326 P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:38.764078 10708 tablet.cc:1722] T de80a938a89f471b8b9c0c4b1b4be326 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:38.769606 10775 tablet_copy_client.cc:523] T 8295ea038a88485095544cc608b000c2 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.770651 6734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=70b7076aa6314855b03666d69272e9fd] alter complete (version 3)
I20240503 01:59:38.771323 10775 tablet_bootstrap.cc:492] T 8295ea038a88485095544cc608b000c2 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:38.800678 10768 tablet_bootstrap.cc:492] T c029253a832e46a48ae16691fc044502 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.801126 10768 tablet_bootstrap.cc:492] T c029253a832e46a48ae16691fc044502 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap complete.
I20240503 01:59:38.801371 10768 ts_tablet_manager.cc:1367] T c029253a832e46a48ae16691fc044502 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.042s user 0.026s sys 0.002s
I20240503 01:59:38.801960 10768 raft_consensus.cc:348] T c029253a832e46a48ae16691fc044502 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:38.802182 10768 raft_consensus.cc:729] T c029253a832e46a48ae16691fc044502 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:38.802430 10768 consensus_queue.cc:260] T c029253a832e46a48ae16691fc044502 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:38.803421 10768 ts_tablet_manager.cc:1398] T c029253a832e46a48ae16691fc044502 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:38.804591 7295 tablet_copy_service.cc:342] P 1d0e738c34af40fa9950c6f086619b3a: Request end of tablet copy session b7062fbbc2b54fff9e99ec4ea5562c76-c029253a832e46a48ae16691fc044502 received from {username='slave'} at 127.0.99.135:51921
I20240503 01:59:38.804679 7295 tablet_copy_service.cc:433] P 1d0e738c34af40fa9950c6f086619b3a: ending tablet copy session b7062fbbc2b54fff9e99ec4ea5562c76-c029253a832e46a48ae16691fc044502 on tablet c029253a832e46a48ae16691fc044502 with peer b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 01:59:38.811481 10775 tablet_bootstrap.cc:492] T 8295ea038a88485095544cc608b000c2 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.811801 10775 tablet_bootstrap.cc:492] T 8295ea038a88485095544cc608b000c2 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap complete.
I20240503 01:59:38.811996 10775 ts_tablet_manager.cc:1367] T 8295ea038a88485095544cc608b000c2 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.041s user 0.024s sys 0.004s
I20240503 01:59:38.812436 10775 raft_consensus.cc:348] T 8295ea038a88485095544cc608b000c2 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:38.812562 10775 raft_consensus.cc:729] T 8295ea038a88485095544cc608b000c2 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:38.812727 10775 consensus_queue.cc:260] T 8295ea038a88485095544cc608b000c2 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:38.813021 10775 ts_tablet_manager.cc:1398] T 8295ea038a88485095544cc608b000c2 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:38.814644 7165 tablet_copy_service.cc:342] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Request end of tablet copy session b7062fbbc2b54fff9e99ec4ea5562c76-8295ea038a88485095544cc608b000c2 received from {username='slave'} at 127.0.99.135:42797
I20240503 01:59:38.814741 7165 tablet_copy_service.cc:433] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: ending tablet copy session b7062fbbc2b54fff9e99ec4ea5562c76-8295ea038a88485095544cc608b000c2 on tablet 8295ea038a88485095544cc608b000c2 with peer b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 01:59:38.814874 6740 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240503 01:59:38.814993 6740 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240503 01:59:38.817669 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 4 replicas of tablet 49b580d648014ce899fee8b3b329c47d
I20240503 01:59:38.818076 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 4 replicas of tablet ccbd932e20b44dc5aceb86c86d7fda6b
I20240503 01:59:38.818280 6993 tablet_service.cc:1514] Processing DeleteTablet for tablet 49b580d648014ce899fee8b3b329c47d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:38 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:38.818290 7254 tablet_service.cc:1514] Processing DeleteTablet for tablet 49b580d648014ce899fee8b3b329c47d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:38 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:38.818377 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 684d38c6115a4d22855bb90b0b784a1f
I20240503 01:59:38.818480 10471 tablet_replica.cc:330] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:38.818487 10468 tablet_replica.cc:330] T 49b580d648014ce899fee8b3b329c47d P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:38.818656 10468 raft_consensus.cc:2229] T 49b580d648014ce899fee8b3b329c47d P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.818693 10471 raft_consensus.cc:2229] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:38.818818 10468 raft_consensus.cc:2258] T 49b580d648014ce899fee8b3b329c47d P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.818962 10471 raft_consensus.cc:2258] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.819382 10468 ts_tablet_manager.cc:1875] T 49b580d648014ce899fee8b3b329c47d P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:38.819458 10471 ts_tablet_manager.cc:1875] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:38.819551 9604 tablet_service.cc:1514] Processing DeleteTablet for tablet 49b580d648014ce899fee8b3b329c47d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:38 UTC) from {username='slave'} at 127.0.0.1:58562
I20240503 01:59:38.820123 7255 tablet_service.cc:1514] Processing DeleteTablet for tablet 684d38c6115a4d22855bb90b0b784a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:38 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:38.820461 7254 tablet_service.cc:1514] Processing DeleteTablet for tablet ccbd932e20b44dc5aceb86c86d7fda6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:38 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:38.821074 7125 tablet_service.cc:1514] Processing DeleteTablet for tablet ccbd932e20b44dc5aceb86c86d7fda6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:38 UTC) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:38.821250 10470 tablet_replica.cc:330] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:38.821427 10470 raft_consensus.cc:2229] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:38.821655 10468 ts_tablet_manager.cc:1888] T 49b580d648014ce899fee8b3b329c47d P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:38.821655 10471 ts_tablet_manager.cc:1888] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:38.821724 10468 log.cc:1196] T 49b580d648014ce899fee8b3b329c47d P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/49b580d648014ce899fee8b3b329c47d
I20240503 01:59:38.821724 10471 log.cc:1196] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/49b580d648014ce899fee8b3b329c47d
I20240503 01:59:38.821743 10470 raft_consensus.cc:2258] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.822315 10468 ts_tablet_manager.cc:1909] T 49b580d648014ce899fee8b3b329c47d P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:38.822324 10470 ts_tablet_manager.cc:1875] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:38.822466 9342 tablet_service.cc:1514] Processing DeleteTablet for tablet ccbd932e20b44dc5aceb86c86d7fda6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:38 UTC) from {username='slave'} at 127.0.0.1:36040
I20240503 01:59:38.823117 6994 tablet_service.cc:1514] Processing DeleteTablet for tablet 684d38c6115a4d22855bb90b0b784a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:38 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:38.823117 6993 tablet_service.cc:1514] Processing DeleteTablet for tablet ccbd932e20b44dc5aceb86c86d7fda6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:38 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:38.823509 10471 ts_tablet_manager.cc:1909] T 49b580d648014ce899fee8b3b329c47d P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:38.823678 10468 tablet_replica.cc:330] T 684d38c6115a4d22855bb90b0b784a1f P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:38.823864 10468 raft_consensus.cc:2229] T 684d38c6115a4d22855bb90b0b784a1f P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.824043 10468 raft_consensus.cc:2258] T 684d38c6115a4d22855bb90b0b784a1f P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.824045 10668 consensus_queue.cc:1035] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:38.824541 10471 tablet_replica.cc:330] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:38.824613 10468 ts_tablet_manager.cc:1875] T 684d38c6115a4d22855bb90b0b784a1f P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:38.824703 10471 raft_consensus.cc:2229] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:38.824759 10470 ts_tablet_manager.cc:1888] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:38.824918 10471 raft_consensus.cc:2258] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.825408 10471 ts_tablet_manager.cc:1875] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:38.826334 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet 49b580d648014ce899fee8b3b329c47d (table rebalance_test_table_10 [id=f8bdc24361c34c13ac0d377e11dc39ae]) successfully deleted
I20240503 01:59:38.826416 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet 49b580d648014ce899fee8b3b329c47d (table rebalance_test_table_10 [id=f8bdc24361c34c13ac0d377e11dc39ae]) successfully deleted
I20240503 01:59:38.826941 10468 ts_tablet_manager.cc:1888] T 684d38c6115a4d22855bb90b0b784a1f P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:38.826989 10470 log.cc:1196] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/ccbd932e20b44dc5aceb86c86d7fda6b
I20240503 01:59:38.826984 10668 consensus_queue.cc:1035] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:38.827082 10468 log.cc:1196] T 684d38c6115a4d22855bb90b0b784a1f P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/684d38c6115a4d22855bb90b0b784a1f
I20240503 01:59:38.827358 10468 ts_tablet_manager.cc:1909] T 684d38c6115a4d22855bb90b0b784a1f P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:38.827754 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet 49b580d648014ce899fee8b3b329c47d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:38 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:38.827805 6863 tablet_service.cc:1514] Processing DeleteTablet for tablet 684d38c6115a4d22855bb90b0b784a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:38 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:38.828068 10468 tablet_replica.cc:330] T ccbd932e20b44dc5aceb86c86d7fda6b P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:38.828171 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet 684d38c6115a4d22855bb90b0b784a1f (table rebalance_test_table_10 [id=f8bdc24361c34c13ac0d377e11dc39ae]) successfully deleted
I20240503 01:59:38.828235 10468 raft_consensus.cc:2229] T ccbd932e20b44dc5aceb86c86d7fda6b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.828413 10468 raft_consensus.cc:2258] T ccbd932e20b44dc5aceb86c86d7fda6b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.828469 9955 tablet_replica.cc:330] T 49b580d648014ce899fee8b3b329c47d P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:38.828608 9955 raft_consensus.cc:2229] T 49b580d648014ce899fee8b3b329c47d P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.828794 9955 raft_consensus.cc:2258] T 49b580d648014ce899fee8b3b329c47d P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.829092 10468 ts_tablet_manager.cc:1875] T ccbd932e20b44dc5aceb86c86d7fda6b P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:38.829300 9955 ts_tablet_manager.cc:1875] T 49b580d648014ce899fee8b3b329c47d P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:38.831284 10468 ts_tablet_manager.cc:1888] T ccbd932e20b44dc5aceb86c86d7fda6b P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:38.831421 10468 log.cc:1196] T ccbd932e20b44dc5aceb86c86d7fda6b P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/ccbd932e20b44dc5aceb86c86d7fda6b
I20240503 01:59:38.831540 9955 ts_tablet_manager.cc:1888] T 49b580d648014ce899fee8b3b329c47d P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:38.831638 9955 log.cc:1196] T 49b580d648014ce899fee8b3b329c47d P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/49b580d648014ce899fee8b3b329c47d
I20240503 01:59:38.831756 10468 ts_tablet_manager.cc:1909] T ccbd932e20b44dc5aceb86c86d7fda6b P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:38.831933 9955 ts_tablet_manager.cc:1909] T 49b580d648014ce899fee8b3b329c47d P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:38.832170 10470 ts_tablet_manager.cc:1909] T ccbd932e20b44dc5aceb86c86d7fda6b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting consensus metadata
I20240503 01:59:38.832579 9955 tablet_replica.cc:330] T 684d38c6115a4d22855bb90b0b784a1f P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:38.832602 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet ccbd932e20b44dc5aceb86c86d7fda6b (table rebalance_test_table_10 [id=f8bdc24361c34c13ac0d377e11dc39ae]) successfully deleted
I20240503 01:59:38.832712 9955 raft_consensus.cc:2229] T 684d38c6115a4d22855bb90b0b784a1f P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.832805 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet ccbd932e20b44dc5aceb86c86d7fda6b (table rebalance_test_table_10 [id=f8bdc24361c34c13ac0d377e11dc39ae]) successfully deleted
I20240503 01:59:38.832851 9955 raft_consensus.cc:2258] T 684d38c6115a4d22855bb90b0b784a1f P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.832722 10471 ts_tablet_manager.cc:1888] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:38.832958 10471 log.cc:1196] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/684d38c6115a4d22855bb90b0b784a1f
I20240503 01:59:38.833365 9955 ts_tablet_manager.cc:1875] T 684d38c6115a4d22855bb90b0b784a1f P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:38.833559 10471 ts_tablet_manager.cc:1909] T 684d38c6115a4d22855bb90b0b784a1f P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:38.834283 10471 tablet_replica.cc:330] T ccbd932e20b44dc5aceb86c86d7fda6b P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:38.834403 10471 raft_consensus.cc:2229] T ccbd932e20b44dc5aceb86c86d7fda6b P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.834510 10471 raft_consensus.cc:2258] T ccbd932e20b44dc5aceb86c86d7fda6b P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.834561 10804 tablet_replica.cc:330] T 49b580d648014ce899fee8b3b329c47d P b7062fbbc2b54fff9e99ec4ea5562c76: stopping tablet replica
I20240503 01:59:38.835124 10805 tablet_replica.cc:330] T ccbd932e20b44dc5aceb86c86d7fda6b P 12e8c683db054df3b75c6285b98a5ad8: stopping tablet replica
I20240503 01:59:38.835296 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet 684d38c6115a4d22855bb90b0b784a1f (table rebalance_test_table_10 [id=f8bdc24361c34c13ac0d377e11dc39ae]) successfully deleted
I20240503 01:59:38.835357 10805 raft_consensus.cc:2229] T ccbd932e20b44dc5aceb86c86d7fda6b P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.835477 10805 raft_consensus.cc:2258] T ccbd932e20b44dc5aceb86c86d7fda6b P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.835744 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 49b580d648014ce899fee8b3b329c47d (table rebalance_test_table_10 [id=f8bdc24361c34c13ac0d377e11dc39ae]) successfully deleted
I20240503 01:59:38.836061 10805 ts_tablet_manager.cc:1875] T ccbd932e20b44dc5aceb86c86d7fda6b P 12e8c683db054df3b75c6285b98a5ad8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:38.836576 10471 ts_tablet_manager.cc:1875] T ccbd932e20b44dc5aceb86c86d7fda6b P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:38.834901 10804 raft_consensus.cc:2229] T 49b580d648014ce899fee8b3b329c47d P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.837641 10804 raft_consensus.cc:2258] T 49b580d648014ce899fee8b3b329c47d P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.838606 10804 ts_tablet_manager.cc:1875] T 49b580d648014ce899fee8b3b329c47d P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:38.839449 10471 ts_tablet_manager.cc:1888] T ccbd932e20b44dc5aceb86c86d7fda6b P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:38.839533 10471 log.cc:1196] T ccbd932e20b44dc5aceb86c86d7fda6b P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/ccbd932e20b44dc5aceb86c86d7fda6b
I20240503 01:59:38.839783 10471 ts_tablet_manager.cc:1909] T ccbd932e20b44dc5aceb86c86d7fda6b P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:38.840451 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet ccbd932e20b44dc5aceb86c86d7fda6b (table rebalance_test_table_10 [id=f8bdc24361c34c13ac0d377e11dc39ae]) successfully deleted
I20240503 01:59:38.841235 9955 ts_tablet_manager.cc:1888] T 684d38c6115a4d22855bb90b0b784a1f P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:38.841364 9955 log.cc:1196] T 684d38c6115a4d22855bb90b0b784a1f P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/684d38c6115a4d22855bb90b0b784a1f
I20240503 01:59:38.841722 9955 ts_tablet_manager.cc:1909] T 684d38c6115a4d22855bb90b0b784a1f P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:38.842557 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 684d38c6115a4d22855bb90b0b784a1f (table rebalance_test_table_10 [id=f8bdc24361c34c13ac0d377e11dc39ae]) successfully deleted
I20240503 01:59:38.845782 10805 ts_tablet_manager.cc:1888] T ccbd932e20b44dc5aceb86c86d7fda6b P 12e8c683db054df3b75c6285b98a5ad8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:38.845887 10805 log.cc:1196] T ccbd932e20b44dc5aceb86c86d7fda6b P 12e8c683db054df3b75c6285b98a5ad8: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/ccbd932e20b44dc5aceb86c86d7fda6b
I20240503 01:59:38.846791 10804 ts_tablet_manager.cc:1888] T 49b580d648014ce899fee8b3b329c47d P b7062fbbc2b54fff9e99ec4ea5562c76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:38.848186 10805 ts_tablet_manager.cc:1909] T ccbd932e20b44dc5aceb86c86d7fda6b P 12e8c683db054df3b75c6285b98a5ad8: Deleting consensus metadata
I20240503 01:59:38.849050 10804 log.cc:1196] T 49b580d648014ce899fee8b3b329c47d P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/49b580d648014ce899fee8b3b329c47d
I20240503 01:59:38.849131 6727 catalog_manager.cc:4830] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): tablet ccbd932e20b44dc5aceb86c86d7fda6b (table rebalance_test_table_10 [id=f8bdc24361c34c13ac0d377e11dc39ae]) successfully deleted
I20240503 01:59:38.849510 10804 ts_tablet_manager.cc:1909] T 49b580d648014ce899fee8b3b329c47d P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting consensus metadata
I20240503 01:59:38.851019 6727 catalog_manager.cc:4830] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): tablet 49b580d648014ce899fee8b3b329c47d (table rebalance_test_table_10 [id=f8bdc24361c34c13ac0d377e11dc39ae]) successfully deleted
I20240503 01:59:38.859665 10829 ts_tablet_manager.cc:902] T db656aaa7b104b3182350de320f67839 P bb5364d730cb4489a0bfc6c270b7450a: Initiating tablet copy from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:38.860364 10829 tablet_copy_client.cc:308] T db656aaa7b104b3182350de320f67839 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41979
I20240503 01:59:38.861344 7165 tablet_copy_service.cc:140] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Received BeginTabletCopySession request for tablet db656aaa7b104b3182350de320f67839 from peer bb5364d730cb4489a0bfc6c270b7450a ({username='slave'} at 127.0.99.134:59853)
I20240503 01:59:38.861478 7165 tablet_copy_service.cc:161] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Beginning new tablet copy session on tablet db656aaa7b104b3182350de320f67839 from peer bb5364d730cb4489a0bfc6c270b7450a at {username='slave'} at 127.0.99.134:59853: session id = bb5364d730cb4489a0bfc6c270b7450a-db656aaa7b104b3182350de320f67839
I20240503 01:59:38.862488 7165 tablet_copy_source_session.cc:205] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:38.864889 10829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db656aaa7b104b3182350de320f67839. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.867834 10829 tablet_copy_client.cc:783] T db656aaa7b104b3182350de320f67839 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:38.868156 10829 tablet_copy_client.cc:647] T db656aaa7b104b3182350de320f67839 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:38.869513 10829 tablet_copy_client.cc:523] T db656aaa7b104b3182350de320f67839 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:38.872280 6735 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:38.872351 10829 tablet_bootstrap.cc:492] T db656aaa7b104b3182350de320f67839 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:38.872421 6735 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:38.877038 6735 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet de80a938a89f471b8b9c0c4b1b4be326
I20240503 01:59:38.878058 9604 tablet_service.cc:1514] Processing DeleteTablet for tablet de80a938a89f471b8b9c0c4b1b4be326 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:38 UTC) from {username='slave'} at 127.0.0.1:58562
I20240503 01:59:38.878257 9473 tablet_service.cc:1514] Processing DeleteTablet for tablet de80a938a89f471b8b9c0c4b1b4be326 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:38 UTC) from {username='slave'} at 127.0.0.1:54678
I20240503 01:59:38.878451 9736 tablet_service.cc:1514] Processing DeleteTablet for tablet de80a938a89f471b8b9c0c4b1b4be326 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:38 UTC) from {username='slave'} at 127.0.0.1:48152
I20240503 01:59:38.879434 10804 tablet_replica.cc:330] T de80a938a89f471b8b9c0c4b1b4be326 P b7062fbbc2b54fff9e99ec4ea5562c76: stopping tablet replica
I20240503 01:59:38.879662 10804 raft_consensus.cc:2229] T de80a938a89f471b8b9c0c4b1b4be326 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.880486 10804 raft_consensus.cc:2258] T de80a938a89f471b8b9c0c4b1b4be326 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.881325 10804 ts_tablet_manager.cc:1875] T de80a938a89f471b8b9c0c4b1b4be326 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:38.883628 10804 ts_tablet_manager.cc:1888] T de80a938a89f471b8b9c0c4b1b4be326 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:38.883782 10804 log.cc:1196] T de80a938a89f471b8b9c0c4b1b4be326 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/de80a938a89f471b8b9c0c4b1b4be326
I20240503 01:59:38.885871 10804 ts_tablet_manager.cc:1909] T de80a938a89f471b8b9c0c4b1b4be326 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting consensus metadata
I20240503 01:59:38.887768 6727 catalog_manager.cc:4830] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): tablet de80a938a89f471b8b9c0c4b1b4be326 (table rebalancer_dynamic_table_new_name [id=70b7076aa6314855b03666d69272e9fd]) successfully deleted
I20240503 01:59:38.898277 10834 tablet_replica.cc:330] T de80a938a89f471b8b9c0c4b1b4be326 P bb5364d730cb4489a0bfc6c270b7450a: stopping tablet replica
I20240503 01:59:38.898933 10834 raft_consensus.cc:2229] T de80a938a89f471b8b9c0c4b1b4be326 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:38.899155 9342 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:38.899201 10834 raft_consensus.cc:2258] T de80a938a89f471b8b9c0c4b1b4be326 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.900522 10834 ts_tablet_manager.cc:1875] T de80a938a89f471b8b9c0c4b1b4be326 P bb5364d730cb4489a0bfc6c270b7450a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:38.904331 10834 ts_tablet_manager.cc:1888] T de80a938a89f471b8b9c0c4b1b4be326 P bb5364d730cb4489a0bfc6c270b7450a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:38.904985 10834 log.cc:1196] T de80a938a89f471b8b9c0c4b1b4be326 P bb5364d730cb4489a0bfc6c270b7450a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/de80a938a89f471b8b9c0c4b1b4be326
I20240503 01:59:38.905445 10834 ts_tablet_manager.cc:1909] T de80a938a89f471b8b9c0c4b1b4be326 P bb5364d730cb4489a0bfc6c270b7450a: Deleting consensus metadata
I20240503 01:59:38.905550 10829 tablet_bootstrap.cc:492] T db656aaa7b104b3182350de320f67839 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:38.905968 10829 tablet_bootstrap.cc:492] T db656aaa7b104b3182350de320f67839 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap complete.
I20240503 01:59:38.906158 10829 ts_tablet_manager.cc:1367] T db656aaa7b104b3182350de320f67839 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.034s user 0.027s sys 0.000s
I20240503 01:59:38.906569 6725 catalog_manager.cc:4830] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): tablet de80a938a89f471b8b9c0c4b1b4be326 (table rebalancer_dynamic_table_new_name [id=70b7076aa6314855b03666d69272e9fd]) successfully deleted
I20240503 01:59:38.906893 10829 raft_consensus.cc:348] T db656aaa7b104b3182350de320f67839 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:38.907100 10829 raft_consensus.cc:729] T db656aaa7b104b3182350de320f67839 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:38.907298 10829 consensus_queue.cc:260] T db656aaa7b104b3182350de320f67839 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:38.907686 10829 ts_tablet_manager.cc:1398] T db656aaa7b104b3182350de320f67839 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:38.908293 7165 tablet_copy_service.cc:342] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Request end of tablet copy session bb5364d730cb4489a0bfc6c270b7450a-db656aaa7b104b3182350de320f67839 received from {username='slave'} at 127.0.99.134:59853
I20240503 01:59:38.908375 7165 tablet_copy_service.cc:433] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: ending tablet copy session bb5364d730cb4489a0bfc6c270b7450a-db656aaa7b104b3182350de320f67839 on tablet db656aaa7b104b3182350de320f67839 with peer bb5364d730cb4489a0bfc6c270b7450a
I20240503 01:59:38.914218 7254 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:38.914701 10835 tablet_replica.cc:330] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191: stopping tablet replica
I20240503 01:59:38.914991 10835 raft_consensus.cc:2229] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:38.915458 10835 raft_consensus.cc:2258] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:38.920248 10835 ts_tablet_manager.cc:1875] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:38.920451 6993 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 01:59:38.922981 10835 ts_tablet_manager.cc:1888] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:38.926105 10835 log.cc:1196] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/de80a938a89f471b8b9c0c4b1b4be326
I20240503 01:59:38.928017 10835 ts_tablet_manager.cc:1909] T de80a938a89f471b8b9c0c4b1b4be326 P db1a0000aaa346f79c0aa4b6c5105191: Deleting consensus metadata
I20240503 01:59:38.928895 6726 catalog_manager.cc:4830] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): tablet de80a938a89f471b8b9c0c4b1b4be326 (table rebalancer_dynamic_table_new_name [id=70b7076aa6314855b03666d69272e9fd]) successfully deleted
I20240503 01:59:38.932883 9604 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:38.941187 6882 raft_consensus.cc:1203] T 31b077072738410f8d7c139ad1b5aa9c P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 01:59:38.943614 6863 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:38.946244 7125 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:38.958130 6735 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:38.960641 9473 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:38.961639 9736 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:38.972355 9473 tablet_service.cc:1467] Processing CreateTablet for tablet 23ae815010364a5586e60b77ff0a600a (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=73d68a2fa3594bc4b64c9e73724c9ae0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:38.972736 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23ae815010364a5586e60b77ff0a600a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.972683 9474 tablet_service.cc:1467] Processing CreateTablet for tablet 4e686819aed5407bacc13b3e3a0d4566 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=73d68a2fa3594bc4b64c9e73724c9ae0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:38.972785 6863 tablet_service.cc:1467] Processing CreateTablet for tablet 23ae815010364a5586e60b77ff0a600a (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=73d68a2fa3594bc4b64c9e73724c9ae0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:38.972930 9474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e686819aed5407bacc13b3e3a0d4566. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.973037 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23ae815010364a5586e60b77ff0a600a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.973243 9736 tablet_service.cc:1467] Processing CreateTablet for tablet 4e686819aed5407bacc13b3e3a0d4566 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=73d68a2fa3594bc4b64c9e73724c9ae0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:38.973485 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e686819aed5407bacc13b3e3a0d4566. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.975435 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 4e686819aed5407bacc13b3e3a0d4566 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=73d68a2fa3594bc4b64c9e73724c9ae0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:38.975677 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e686819aed5407bacc13b3e3a0d4566. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.976325 10850 tablet_bootstrap.cc:492] T 4e686819aed5407bacc13b3e3a0d4566 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:38.976603 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 23ae815010364a5586e60b77ff0a600a (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=73d68a2fa3594bc4b64c9e73724c9ae0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:38.976866 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23ae815010364a5586e60b77ff0a600a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.977272 10850 tablet_bootstrap.cc:654] T 4e686819aed5407bacc13b3e3a0d4566 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:38.977861 7254 tablet_service.cc:1467] Processing CreateTablet for tablet 5b386bc757e44ca588e3ba6ca76c92b6 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=73d68a2fa3594bc4b64c9e73724c9ae0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:38.978116 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b386bc757e44ca588e3ba6ca76c92b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.978920 9604 tablet_service.cc:1467] Processing CreateTablet for tablet 5b386bc757e44ca588e3ba6ca76c92b6 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=73d68a2fa3594bc4b64c9e73724c9ae0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:38.979202 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b386bc757e44ca588e3ba6ca76c92b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.980086 6738 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:38.981647 10854 tablet_bootstrap.cc:492] T 5b386bc757e44ca588e3ba6ca76c92b6 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:38.983701 9736 tablet_service.cc:1467] Processing CreateTablet for tablet 5b386bc757e44ca588e3ba6ca76c92b6 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=73d68a2fa3594bc4b64c9e73724c9ae0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:38.983978 10854 tablet_bootstrap.cc:654] T 5b386bc757e44ca588e3ba6ca76c92b6 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:38.984021 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b386bc757e44ca588e3ba6ca76c92b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:38.985376 10853 tablet_bootstrap.cc:492] T 5b386bc757e44ca588e3ba6ca76c92b6 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:38.986451 10853 tablet_bootstrap.cc:654] T 5b386bc757e44ca588e3ba6ca76c92b6 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:38.988364 10853 tablet_bootstrap.cc:492] T 5b386bc757e44ca588e3ba6ca76c92b6 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:38.988507 10853 ts_tablet_manager.cc:1367] T 5b386bc757e44ca588e3ba6ca76c92b6 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:38.989001 10854 tablet_bootstrap.cc:492] T 5b386bc757e44ca588e3ba6ca76c92b6 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:38.989029 10853 raft_consensus.cc:348] T 5b386bc757e44ca588e3ba6ca76c92b6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:38.989189 10853 raft_consensus.cc:374] T 5b386bc757e44ca588e3ba6ca76c92b6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:38.989238 10853 raft_consensus.cc:729] T 5b386bc757e44ca588e3ba6ca76c92b6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:38.989362 10854 ts_tablet_manager.cc:1367] T 5b386bc757e44ca588e3ba6ca76c92b6 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20240503 01:59:38.989382 10853 consensus_queue.cc:260] T 5b386bc757e44ca588e3ba6ca76c92b6 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:38.989908 10854 raft_consensus.cc:348] T 5b386bc757e44ca588e3ba6ca76c92b6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:38.990083 10852 tablet_bootstrap.cc:492] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:38.990450 10853 ts_tablet_manager.cc:1398] T 5b386bc757e44ca588e3ba6ca76c92b6 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:38.991173 10854 raft_consensus.cc:374] T 5b386bc757e44ca588e3ba6ca76c92b6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:38.991564 10854 raft_consensus.cc:729] T 5b386bc757e44ca588e3ba6ca76c92b6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:38.991801 10854 consensus_queue.cc:260] T 5b386bc757e44ca588e3ba6ca76c92b6 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:38.992727 10852 tablet_bootstrap.cc:654] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:38.994136 10854 ts_tablet_manager.cc:1398] T 5b386bc757e44ca588e3ba6ca76c92b6 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240503 01:59:38.994886 10852 tablet_bootstrap.cc:492] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:38.994937 10849 tablet_bootstrap.cc:492] T 23ae815010364a5586e60b77ff0a600a P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:38.995213 10852 ts_tablet_manager.cc:1367] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:59:38.995903 10849 tablet_bootstrap.cc:654] T 23ae815010364a5586e60b77ff0a600a P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:38.995939 10852 raft_consensus.cc:348] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:38.996150 10852 raft_consensus.cc:374] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:38.996237 10852 raft_consensus.cc:729] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:38.996414 10852 consensus_queue.cc:260] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:38.997308 10852 ts_tablet_manager.cc:1398] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:38.999472 10848 tablet_bootstrap.cc:492] T 23ae815010364a5586e60b77ff0a600a P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:39.000747 10848 tablet_bootstrap.cc:654] T 23ae815010364a5586e60b77ff0a600a P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:39.002394 9604 tablet_service.cc:1467] Processing CreateTablet for tablet 9c589cec02484767aff92eb183f264c4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6901d9bbd44b491e820471118eab2b38]), partition=
I20240503 01:59:39.002466 9473 tablet_service.cc:1467] Processing CreateTablet for tablet 9c589cec02484767aff92eb183f264c4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6901d9bbd44b491e820471118eab2b38]), partition=
I20240503 01:59:39.002602 10848 tablet_bootstrap.cc:492] T 23ae815010364a5586e60b77ff0a600a P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:39.002646 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c589cec02484767aff92eb183f264c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:39.002689 10848 ts_tablet_manager.cc:1367] T 23ae815010364a5586e60b77ff0a600a P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:39.002713 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c589cec02484767aff92eb183f264c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:39.003171 10848 raft_consensus.cc:348] T 23ae815010364a5586e60b77ff0a600a P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:39.003273 10848 raft_consensus.cc:374] T 23ae815010364a5586e60b77ff0a600a P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:39.003300 10848 raft_consensus.cc:729] T 23ae815010364a5586e60b77ff0a600a P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:39.003652 10848 consensus_queue.cc:260] T 23ae815010364a5586e60b77ff0a600a P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:39.004398 10848 ts_tablet_manager.cc:1398] T 23ae815010364a5586e60b77ff0a600a P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:39.004590 10848 tablet_bootstrap.cc:492] T 4e686819aed5407bacc13b3e3a0d4566 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:39.005263 10854 tablet_bootstrap.cc:492] T 9c589cec02484767aff92eb183f264c4 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:39.005604 10848 tablet_bootstrap.cc:654] T 4e686819aed5407bacc13b3e3a0d4566 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:39.006151 10854 tablet_bootstrap.cc:654] T 9c589cec02484767aff92eb183f264c4 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:39.006765 10848 tablet_bootstrap.cc:492] T 4e686819aed5407bacc13b3e3a0d4566 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:39.006841 10848 ts_tablet_manager.cc:1367] T 4e686819aed5407bacc13b3e3a0d4566 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:39.007310 10848 raft_consensus.cc:348] T 4e686819aed5407bacc13b3e3a0d4566 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:39.007418 10850 tablet_bootstrap.cc:492] T 4e686819aed5407bacc13b3e3a0d4566 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:39.007426 10848 raft_consensus.cc:374] T 4e686819aed5407bacc13b3e3a0d4566 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:39.007469 10848 raft_consensus.cc:729] T 4e686819aed5407bacc13b3e3a0d4566 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:39.007511 10850 ts_tablet_manager.cc:1367] T 4e686819aed5407bacc13b3e3a0d4566 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.000s
I20240503 01:59:39.007607 10848 consensus_queue.cc:260] T 4e686819aed5407bacc13b3e3a0d4566 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:39.007869 10848 ts_tablet_manager.cc:1398] T 4e686819aed5407bacc13b3e3a0d4566 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:39.007889 10850 raft_consensus.cc:348] T 4e686819aed5407bacc13b3e3a0d4566 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:39.007983 10848 tablet_bootstrap.cc:492] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:39.007997 10850 raft_consensus.cc:374] T 4e686819aed5407bacc13b3e3a0d4566 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:39.008025 10850 raft_consensus.cc:729] T 4e686819aed5407bacc13b3e3a0d4566 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:39.008147 10850 consensus_queue.cc:260] T 4e686819aed5407bacc13b3e3a0d4566 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:39.008409 10850 ts_tablet_manager.cc:1398] T 4e686819aed5407bacc13b3e3a0d4566 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:39.008553 10850 tablet_bootstrap.cc:492] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:39.008745 10848 tablet_bootstrap.cc:654] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:39.009341 10850 tablet_bootstrap.cc:654] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:39.010243 7254 tablet_service.cc:1467] Processing CreateTablet for tablet 9c589cec02484767aff92eb183f264c4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6901d9bbd44b491e820471118eab2b38]), partition=
I20240503 01:59:39.010511 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c589cec02484767aff92eb183f264c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:39.011699 10389 consensus_queue.cc:1035] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:39.012316 10850 tablet_bootstrap.cc:492] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:39.012400 10850 ts_tablet_manager.cc:1367] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:39.012851 10850 raft_consensus.cc:348] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:39.012961 10850 raft_consensus.cc:374] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:39.013005 10850 raft_consensus.cc:729] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:39.013120 10850 consensus_queue.cc:260] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:39.013311 10853 tablet_bootstrap.cc:492] T 9c589cec02484767aff92eb183f264c4 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:39.013428 10850 ts_tablet_manager.cc:1398] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:39.014272 10853 tablet_bootstrap.cc:654] T 9c589cec02484767aff92eb183f264c4 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:39.015658 10848 tablet_bootstrap.cc:492] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:39.015851 10848 ts_tablet_manager.cc:1367] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 01:59:39.015934 10580 consensus_queue.cc:1035] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:39.016412 10848 raft_consensus.cc:348] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:39.016665 10848 raft_consensus.cc:374] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:39.016767 10848 raft_consensus.cc:729] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:39.017002 10848 consensus_queue.cc:260] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:39.017485 10848 ts_tablet_manager.cc:1398] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:39.018884 10853 tablet_bootstrap.cc:492] T 9c589cec02484767aff92eb183f264c4 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:39.018977 10853 ts_tablet_manager.cc:1367] T 9c589cec02484767aff92eb183f264c4 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:39.019446 10853 raft_consensus.cc:348] T 9c589cec02484767aff92eb183f264c4 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:39.019567 10853 raft_consensus.cc:374] T 9c589cec02484767aff92eb183f264c4 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:39.019609 10853 raft_consensus.cc:729] T 9c589cec02484767aff92eb183f264c4 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:39.019729 10853 consensus_queue.cc:260] T 9c589cec02484767aff92eb183f264c4 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:39.020031 10853 ts_tablet_manager.cc:1398] T 9c589cec02484767aff92eb183f264c4 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:39.028723 10854 tablet_bootstrap.cc:492] T 9c589cec02484767aff92eb183f264c4 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:39.028862 10854 ts_tablet_manager.cc:1367] T 9c589cec02484767aff92eb183f264c4 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240503 01:59:39.029465 10854 raft_consensus.cc:348] T 9c589cec02484767aff92eb183f264c4 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:39.029596 10854 raft_consensus.cc:374] T 9c589cec02484767aff92eb183f264c4 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:39.029631 10854 raft_consensus.cc:729] T 9c589cec02484767aff92eb183f264c4 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:39.029783 10849 tablet_bootstrap.cc:492] T 23ae815010364a5586e60b77ff0a600a P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:39.029758 10854 consensus_queue.cc:260] T 9c589cec02484767aff92eb183f264c4 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:39.029876 10849 ts_tablet_manager.cc:1367] T 23ae815010364a5586e60b77ff0a600a P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.000s
I20240503 01:59:39.030072 10854 ts_tablet_manager.cc:1398] T 9c589cec02484767aff92eb183f264c4 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:39.030375 10849 raft_consensus.cc:348] T 23ae815010364a5586e60b77ff0a600a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:39.030483 10849 raft_consensus.cc:374] T 23ae815010364a5586e60b77ff0a600a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:39.030572 10849 raft_consensus.cc:729] T 23ae815010364a5586e60b77ff0a600a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:39.030704 10849 consensus_queue.cc:260] T 23ae815010364a5586e60b77ff0a600a P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:39.030969 10849 ts_tablet_manager.cc:1398] T 23ae815010364a5586e60b77ff0a600a P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:39.031177 10849 tablet_bootstrap.cc:492] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:39.032402 10849 tablet_bootstrap.cc:654] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:39.035254 10849 tablet_bootstrap.cc:492] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:39.035338 10849 ts_tablet_manager.cc:1367] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:39.035724 10849 raft_consensus.cc:348] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:39.036015 10849 raft_consensus.cc:374] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:39.036139 10849 raft_consensus.cc:729] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:39.036365 10849 consensus_queue.cc:260] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:39.037550 10849 ts_tablet_manager.cc:1398] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:39.103404 10869 raft_consensus.cc:482] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:39.103530 10869 raft_consensus.cc:504] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:39.104004 10869 leader_election.cc:288] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:39.104735 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c589cec02484767aff92eb183f264c4" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:39.104897 7274 raft_consensus.cc:2454] T 9c589cec02484767aff92eb183f264c4 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:39.105554 9428 leader_election.cc:302] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:39.105756 10869 raft_consensus.cc:2781] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:39.105818 10869 raft_consensus.cc:482] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:39.105854 10869 raft_consensus.cc:3037] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:39.106976 10869 raft_consensus.cc:504] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:39.107345 10869 leader_election.cc:288] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:39.107858 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c589cec02484767aff92eb183f264c4" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:39.108109 7274 raft_consensus.cc:3037] T 9c589cec02484767aff92eb183f264c4 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:39.109114 7274 raft_consensus.cc:2454] T 9c589cec02484767aff92eb183f264c4 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:39.109783 9428 leader_election.cc:302] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:39.109946 10869 raft_consensus.cc:2781] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:39.110026 10869 raft_consensus.cc:686] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Becoming Leader. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Running, Role: LEADER
I20240503 01:59:39.110153 10869 consensus_queue.cc:237] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:39.112219 6740 catalog_manager.cc:5484] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: term changed from 0 to 1, leader changed from <none> to bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:39.122447 10273 raft_consensus.cc:482] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:39.122615 10273 raft_consensus.cc:504] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:39.123070 10273 leader_election.cc:288] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:39.123557 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e686819aed5407bacc13b3e3a0d4566" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:39.123737 9756 raft_consensus.cc:2454] T 4e686819aed5407bacc13b3e3a0d4566 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:39.123876 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e686819aed5407bacc13b3e3a0d4566" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:39.124019 9494 raft_consensus.cc:2454] T 4e686819aed5407bacc13b3e3a0d4566 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:39.124231 6817 leader_election.cc:302] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:39.124761 10273 raft_consensus.cc:2781] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:39.124819 10273 raft_consensus.cc:482] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:39.124790 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c589cec02484767aff92eb183f264c4" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:39.124864 10273 raft_consensus.cc:3037] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:39.124816 9626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c589cec02484767aff92eb183f264c4" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:39.124935 9625 raft_consensus.cc:2454] T 9c589cec02484767aff92eb183f264c4 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:39.126026 10273 raft_consensus.cc:504] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:39.126415 10273 leader_election.cc:288] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Requested vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:39.126916 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e686819aed5407bacc13b3e3a0d4566" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:39.127041 9494 raft_consensus.cc:3037] T 4e686819aed5407bacc13b3e3a0d4566 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:39.127786 9494 raft_consensus.cc:2454] T 4e686819aed5407bacc13b3e3a0d4566 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:39.128062 6816 leader_election.cc:302] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:39.128219 10273 raft_consensus.cc:2781] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:39.128232 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e686819aed5407bacc13b3e3a0d4566" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:39.128304 10273 raft_consensus.cc:686] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 01:59:39.128362 9756 raft_consensus.cc:3037] T 4e686819aed5407bacc13b3e3a0d4566 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:39.128445 10273 consensus_queue.cc:237] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:39.129421 9756 raft_consensus.cc:2454] T 4e686819aed5407bacc13b3e3a0d4566 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:39.131222 6738 catalog_manager.cc:5484] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:39.147953 10078 raft_consensus.cc:482] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:39.148108 10078 raft_consensus.cc:504] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:39.148536 10078 leader_election.cc:288] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:39.149163 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ae815010364a5586e60b77ff0a600a" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:39.149185 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ae815010364a5586e60b77ff0a600a" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:39.149317 6882 raft_consensus.cc:2454] T 23ae815010364a5586e60b77ff0a600a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:39.149325 9494 raft_consensus.cc:2454] T 23ae815010364a5586e60b77ff0a600a P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:39.149646 9299 leader_election.cc:302] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:39.150023 10078 raft_consensus.cc:2781] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:39.150081 10078 raft_consensus.cc:482] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:39.150120 10078 raft_consensus.cc:3037] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:39.151274 10078 raft_consensus.cc:504] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:39.151695 10078 leader_election.cc:288] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Requested vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:39.152025 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ae815010364a5586e60b77ff0a600a" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:39.152153 6882 raft_consensus.cc:3037] T 23ae815010364a5586e60b77ff0a600a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:39.152468 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ae815010364a5586e60b77ff0a600a" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:39.152601 9494 raft_consensus.cc:3037] T 23ae815010364a5586e60b77ff0a600a P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:39.153198 6882 raft_consensus.cc:2454] T 23ae815010364a5586e60b77ff0a600a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:39.153326 9494 raft_consensus.cc:2454] T 23ae815010364a5586e60b77ff0a600a P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:39.153563 9299 leader_election.cc:302] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:39.153791 10078 raft_consensus.cc:2781] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:39.154004 10078 raft_consensus.cc:686] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Running, Role: LEADER
I20240503 01:59:39.154204 10078 consensus_queue.cc:237] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:39.156281 6741 catalog_manager.cc:5484] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:39.218056 9625 raft_consensus.cc:1203] T 8295ea038a88485095544cc608b000c2 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:39.221617 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:39.226014 9625 raft_consensus.cc:3037] T 9c589cec02484767aff92eb183f264c4 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:39.226075 7274 raft_consensus.cc:1261] T 9c589cec02484767aff92eb183f264c4 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:39.227284 9625 raft_consensus.cc:1261] T 9c589cec02484767aff92eb183f264c4 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:39.227648 10359 consensus_queue.cc:1035] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:39.227838 10869 consensus_queue.cc:1035] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:39.230370 6734 catalog_manager.cc:5576] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087) does not have the latest schema for tablet 9c589cec02484767aff92eb183f264c4 (table rebalancer_dynamic_table [id=6901d9bbd44b491e820471118eab2b38]). Expected version 1 got 0
I20240503 01:59:39.230615 10708 tablet.cc:1722] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:39.230966 10708 tablet.cc:1620] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:39.232657 10709 tablet.cc:1722] T 9c589cec02484767aff92eb183f264c4 P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:39.232947 10709 tablet.cc:1620] T 9c589cec02484767aff92eb183f264c4 P b7062fbbc2b54fff9e99ec4ea5562c76: MemRowSet was empty: no flush needed.
I20240503 01:59:39.233302 6741 catalog_manager.cc:5576] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271) does not have the latest schema for tablet 9c589cec02484767aff92eb183f264c4 (table rebalancer_dynamic_table [id=6901d9bbd44b491e820471118eab2b38]). Expected version 1 got 0
I20240503 01:59:39.235639 10921 tablet.cc:1722] T 9c589cec02484767aff92eb183f264c4 P 1d0e738c34af40fa9950c6f086619b3a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:39.236044 10921 tablet.cc:1620] T 9c589cec02484767aff92eb183f264c4 P 1d0e738c34af40fa9950c6f086619b3a: MemRowSet was empty: no flush needed.
I20240503 01:59:39.236840 6734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=6901d9bbd44b491e820471118eab2b38] alter complete (version 1)
I20240503 01:59:39.238267 6741 catalog_manager.cc:5576] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271) does not have the latest schema for tablet 9c589cec02484767aff92eb183f264c4 (table rebalancer_dynamic_table [id=6901d9bbd44b491e820471118eab2b38]). Expected version 1 got 0
I20240503 01:59:39.294503 9625 raft_consensus.cc:1203] T c029253a832e46a48ae16691fc044502 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:39.317234 9970 raft_consensus.cc:482] T 5b386bc757e44ca588e3ba6ca76c92b6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:39.317399 9970 raft_consensus.cc:504] T 5b386bc757e44ca588e3ba6ca76c92b6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:39.317903 9970 leader_election.cc:288] T 5b386bc757e44ca588e3ba6ca76c92b6 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:39.318303 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b386bc757e44ca588e3ba6ca76c92b6" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:39.318367 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b386bc757e44ca588e3ba6ca76c92b6" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:39.318460 9756 raft_consensus.cc:2454] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 01:59:39.318492 7274 raft_consensus.cc:2454] T 5b386bc757e44ca588e3ba6ca76c92b6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 01:59:39.318837 9560 leader_election.cc:302] T 5b386bc757e44ca588e3ba6ca76c92b6 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:39.319026 9970 raft_consensus.cc:2781] T 5b386bc757e44ca588e3ba6ca76c92b6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:39.319089 9970 raft_consensus.cc:482] T 5b386bc757e44ca588e3ba6ca76c92b6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:39.319144 9970 raft_consensus.cc:3037] T 5b386bc757e44ca588e3ba6ca76c92b6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:39.319985 9970 raft_consensus.cc:504] T 5b386bc757e44ca588e3ba6ca76c92b6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:39.320317 9970 leader_election.cc:288] T 5b386bc757e44ca588e3ba6ca76c92b6 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Requested vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:39.320698 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b386bc757e44ca588e3ba6ca76c92b6" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:39.320766 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b386bc757e44ca588e3ba6ca76c92b6" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:39.320849 9756 raft_consensus.cc:3037] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:39.320878 7274 raft_consensus.cc:3037] T 5b386bc757e44ca588e3ba6ca76c92b6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:39.321929 7274 raft_consensus.cc:2454] T 5b386bc757e44ca588e3ba6ca76c92b6 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 01:59:39.321929 9756 raft_consensus.cc:2454] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 01:59:39.322253 9560 leader_election.cc:302] T 5b386bc757e44ca588e3ba6ca76c92b6 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:39.322495 9970 raft_consensus.cc:2781] T 5b386bc757e44ca588e3ba6ca76c92b6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:39.322625 9970 raft_consensus.cc:686] T 5b386bc757e44ca588e3ba6ca76c92b6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 LEADER]: Becoming Leader. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Running, Role: LEADER
I20240503 01:59:39.322798 9970 consensus_queue.cc:237] T 5b386bc757e44ca588e3ba6ca76c92b6 P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:39.324894 6740 catalog_manager.cc:5484] T 5b386bc757e44ca588e3ba6ca76c92b6 P b7062fbbc2b54fff9e99ec4ea5562c76 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:39.331254 6741 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240503 01:59:39.331482 6741 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240503 01:59:39.334080 6741 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 1eca51b6a18e4718b43f318ed3147338
I20240503 01:59:39.334786 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet 1eca51b6a18e4718b43f318ed3147338 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:39 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:39.334798 7125 tablet_service.cc:1514] Processing DeleteTablet for tablet 1eca51b6a18e4718b43f318ed3147338 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:39 UTC) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:39.334800 6993 tablet_service.cc:1514] Processing DeleteTablet for tablet 1eca51b6a18e4718b43f318ed3147338 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:39 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:39.334977 10471 tablet_replica.cc:330] T 1eca51b6a18e4718b43f318ed3147338 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:39.335008 9955 tablet_replica.cc:330] T 1eca51b6a18e4718b43f318ed3147338 P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:39.335140 10471 raft_consensus.cc:2229] T 1eca51b6a18e4718b43f318ed3147338 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.335142 9955 raft_consensus.cc:2229] T 1eca51b6a18e4718b43f318ed3147338 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.335317 10471 raft_consensus.cc:2258] T 1eca51b6a18e4718b43f318ed3147338 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.335297 9955 raft_consensus.cc:2258] T 1eca51b6a18e4718b43f318ed3147338 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.335860 10471 ts_tablet_manager.cc:1875] T 1eca51b6a18e4718b43f318ed3147338 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:39.336051 9955 ts_tablet_manager.cc:1875] T 1eca51b6a18e4718b43f318ed3147338 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:39.338130 10471 ts_tablet_manager.cc:1888] T 1eca51b6a18e4718b43f318ed3147338 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:39.338205 10471 log.cc:1196] T 1eca51b6a18e4718b43f318ed3147338 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/1eca51b6a18e4718b43f318ed3147338
I20240503 01:59:39.338478 10471 ts_tablet_manager.cc:1909] T 1eca51b6a18e4718b43f318ed3147338 P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:39.339018 10923 tablet_replica.cc:330] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:39.339217 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet 1eca51b6a18e4718b43f318ed3147338 (table rebalance_test_table_12 [id=9e4998e19d924c9a9fd0a10e74520d26]) successfully deleted
I20240503 01:59:39.339288 6741 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet f827ec6764cc4dccba77493b3d1d8ec7
I20240503 01:59:39.339462 10923 raft_consensus.cc:2229] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:39.339767 6993 tablet_service.cc:1514] Processing DeleteTablet for tablet f827ec6764cc4dccba77493b3d1d8ec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:39 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:39.339857 7254 tablet_service.cc:1514] Processing DeleteTablet for tablet f827ec6764cc4dccba77493b3d1d8ec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:39 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:39.339927 10471 tablet_replica.cc:330] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:39.340044 10471 raft_consensus.cc:2229] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:39.340186 9955 ts_tablet_manager.cc:1888] T 1eca51b6a18e4718b43f318ed3147338 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:39.340269 10471 raft_consensus.cc:2258] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.340322 9955 log.cc:1196] T 1eca51b6a18e4718b43f318ed3147338 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/1eca51b6a18e4718b43f318ed3147338
I20240503 01:59:39.340679 9955 ts_tablet_manager.cc:1909] T 1eca51b6a18e4718b43f318ed3147338 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:39.340672 10923 raft_consensus.cc:2258] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.341270 10923 ts_tablet_manager.cc:1875] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:39.341451 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 1eca51b6a18e4718b43f318ed3147338 (table rebalance_test_table_12 [id=9e4998e19d924c9a9fd0a10e74520d26]) successfully deleted
I20240503 01:59:39.341459 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:39.342093 10471 ts_tablet_manager.cc:1875] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:39.344333 10471 ts_tablet_manager.cc:1888] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:39.344523 10471 log.cc:1196] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/f827ec6764cc4dccba77493b3d1d8ec7
I20240503 01:59:39.344436 10923 ts_tablet_manager.cc:1888] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:39.344607 10923 log.cc:1196] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/1eca51b6a18e4718b43f318ed3147338
I20240503 01:59:39.344920 10471 ts_tablet_manager.cc:1909] T f827ec6764cc4dccba77493b3d1d8ec7 P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:39.344960 10923 ts_tablet_manager.cc:1909] T 1eca51b6a18e4718b43f318ed3147338 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting consensus metadata
I20240503 01:59:39.345978 6741 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 789162c6428c49fb9c0edd30960d13c1
I20240503 01:59:39.346107 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet 1eca51b6a18e4718b43f318ed3147338 (table rebalance_test_table_12 [id=9e4998e19d924c9a9fd0a10e74520d26]) successfully deleted
I20240503 01:59:39.346297 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet f827ec6764cc4dccba77493b3d1d8ec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:39 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:39.346689 7254 tablet_service.cc:1514] Processing DeleteTablet for tablet 789162c6428c49fb9c0edd30960d13c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:39 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:39.347000 9955 tablet_replica.cc:330] T f827ec6764cc4dccba77493b3d1d8ec7 P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:39.347141 7125 tablet_service.cc:1514] Processing DeleteTablet for tablet 789162c6428c49fb9c0edd30960d13c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:39 UTC) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:39.347201 9955 raft_consensus.cc:2229] T f827ec6764cc4dccba77493b3d1d8ec7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.347344 9955 raft_consensus.cc:2258] T f827ec6764cc4dccba77493b3d1d8ec7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.347885 9955 ts_tablet_manager.cc:1875] T f827ec6764cc4dccba77493b3d1d8ec7 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:39.348464 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet f827ec6764cc4dccba77493b3d1d8ec7 (table rebalance_test_table_12 [id=9e4998e19d924c9a9fd0a10e74520d26]) successfully deleted
I20240503 01:59:39.349103 6993 tablet_service.cc:1514] Processing DeleteTablet for tablet 789162c6428c49fb9c0edd30960d13c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:39 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:39.351357 10471 tablet_replica.cc:330] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:39.351552 10471 raft_consensus.cc:2229] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:39.351765 10471 raft_consensus.cc:2258] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.351751 10708 tablet.cc:1722] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:39.352105 10708 tablet.cc:1620] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:39.352342 10471 ts_tablet_manager.cc:1875] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:39.355676 10471 ts_tablet_manager.cc:1888] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:39.355849 10471 log.cc:1196] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/789162c6428c49fb9c0edd30960d13c1
I20240503 01:59:39.356155 10471 ts_tablet_manager.cc:1909] T 789162c6428c49fb9c0edd30960d13c1 P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:39.357492 10709 tablet.cc:1722] T 9c589cec02484767aff92eb183f264c4 P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:39.357860 10709 tablet.cc:1620] T 9c589cec02484767aff92eb183f264c4 P b7062fbbc2b54fff9e99ec4ea5562c76: MemRowSet was empty: no flush needed.
I20240503 01:59:39.357954 10921 tablet.cc:1722] T 9c589cec02484767aff92eb183f264c4 P 1d0e738c34af40fa9950c6f086619b3a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:39.358249 10921 tablet.cc:1620] T 9c589cec02484767aff92eb183f264c4 P 1d0e738c34af40fa9950c6f086619b3a: MemRowSet was empty: no flush needed.
I20240503 01:59:39.361148 9342 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:39.362615 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet 789162c6428c49fb9c0edd30960d13c1 (table rebalance_test_table_12 [id=9e4998e19d924c9a9fd0a10e74520d26]) successfully deleted
I20240503 01:59:39.364714 9955 ts_tablet_manager.cc:1888] T f827ec6764cc4dccba77493b3d1d8ec7 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:39.364811 9955 log.cc:1196] T f827ec6764cc4dccba77493b3d1d8ec7 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/f827ec6764cc4dccba77493b3d1d8ec7
I20240503 01:59:39.365085 9955 ts_tablet_manager.cc:1909] T f827ec6764cc4dccba77493b3d1d8ec7 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:39.365813 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet f827ec6764cc4dccba77493b3d1d8ec7 (table rebalance_test_table_12 [id=9e4998e19d924c9a9fd0a10e74520d26]) successfully deleted
I20240503 01:59:39.365850 10923 tablet_replica.cc:330] T 789162c6428c49fb9c0edd30960d13c1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:39.366000 10923 raft_consensus.cc:2229] T 789162c6428c49fb9c0edd30960d13c1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.366241 10924 tablet_replica.cc:330] T f827ec6764cc4dccba77493b3d1d8ec7 P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:39.366423 10923 raft_consensus.cc:2258] T 789162c6428c49fb9c0edd30960d13c1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.366693 10924 raft_consensus.cc:2229] T f827ec6764cc4dccba77493b3d1d8ec7 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.367362 10924 raft_consensus.cc:2258] T f827ec6764cc4dccba77493b3d1d8ec7 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.368033 10923 ts_tablet_manager.cc:1875] T 789162c6428c49fb9c0edd30960d13c1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:39.368510 10924 ts_tablet_manager.cc:1875] T f827ec6764cc4dccba77493b3d1d8ec7 P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:39.374006 6741 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=6901d9bbd44b491e820471118eab2b38] alter complete (version 2)
I20240503 01:59:39.380774 10923 ts_tablet_manager.cc:1888] T 789162c6428c49fb9c0edd30960d13c1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:39.383019 10923 log.cc:1196] T 789162c6428c49fb9c0edd30960d13c1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/789162c6428c49fb9c0edd30960d13c1
I20240503 01:59:39.383880 10924 ts_tablet_manager.cc:1888] T f827ec6764cc4dccba77493b3d1d8ec7 P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:39.387061 9604 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:39.387497 10923 ts_tablet_manager.cc:1909] T 789162c6428c49fb9c0edd30960d13c1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting consensus metadata
I20240503 01:59:39.388550 9494 raft_consensus.cc:1203] T db656aaa7b104b3182350de320f67839 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:39.391131 7125 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:39.392509 6861 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:39.393154 6993 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:39.391252 7254 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:39.394982 9736 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:39.395318 10924 log.cc:1196] T f827ec6764cc4dccba77493b3d1d8ec7 P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/f827ec6764cc4dccba77493b3d1d8ec7
I20240503 01:59:39.395761 10924 ts_tablet_manager.cc:1909] T f827ec6764cc4dccba77493b3d1d8ec7 P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:39.396265 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet 789162c6428c49fb9c0edd30960d13c1 (table rebalance_test_table_12 [id=9e4998e19d924c9a9fd0a10e74520d26]) successfully deleted
I20240503 01:59:39.399510 10924 tablet_replica.cc:330] T 789162c6428c49fb9c0edd30960d13c1 P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:39.399751 10924 raft_consensus.cc:2229] T 789162c6428c49fb9c0edd30960d13c1 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.399977 10924 raft_consensus.cc:2258] T 789162c6428c49fb9c0edd30960d13c1 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.400627 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet f827ec6764cc4dccba77493b3d1d8ec7 (table rebalance_test_table_12 [id=9e4998e19d924c9a9fd0a10e74520d26]) successfully deleted
I20240503 01:59:39.400700 10924 ts_tablet_manager.cc:1875] T 789162c6428c49fb9c0edd30960d13c1 P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:39.405128 10924 ts_tablet_manager.cc:1888] T 789162c6428c49fb9c0edd30960d13c1 P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:39.405369 10924 log.cc:1196] T 789162c6428c49fb9c0edd30960d13c1 P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/789162c6428c49fb9c0edd30960d13c1
I20240503 01:59:39.405786 10924 ts_tablet_manager.cc:1909] T 789162c6428c49fb9c0edd30960d13c1 P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:39.406729 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet 789162c6428c49fb9c0edd30960d13c1 (table rebalance_test_table_12 [id=9e4998e19d924c9a9fd0a10e74520d26]) successfully deleted
I20240503 01:59:39.416986 9473 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:39.465109 7274 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c6833c997d284063955d0b9b09ccb3b7"
dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51624
I20240503 01:59:39.465302 7274 raft_consensus.cc:595] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Received request to transfer leadership
I20240503 01:59:39.473994 6741 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:39.488965 10708 tablet.cc:1722] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:39.491894 10709 tablet.cc:1722] T 9c589cec02484767aff92eb183f264c4 P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:39.495342 10580 raft_consensus.cc:979] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a: : Instructing follower c688f3df56ac4b6d9b0c30fa4cc8ca2a to start an election
I20240503 01:59:39.495457 10580 raft_consensus.cc:1067] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Signalling peer c688f3df56ac4b6d9b0c30fa4cc8ca2a to start an election
I20240503 01:59:39.496335 10921 tablet.cc:1722] T 9c589cec02484767aff92eb183f264c4 P 1d0e738c34af40fa9950c6f086619b3a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:39.498134 7143 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c6833c997d284063955d0b9b09ccb3b7"
dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
from {username='slave'} at 127.0.99.132:38863
I20240503 01:59:39.498319 7143 raft_consensus.cc:482] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 01:59:39.498383 7143 raft_consensus.cc:3037] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:39.499668 6740 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=6901d9bbd44b491e820471118eab2b38] alter complete (version 3)
I20240503 01:59:39.501708 7143 raft_consensus.cc:504] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:39.502310 7143 leader_election.cc:288] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:39.503490 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6833c997d284063955d0b9b09ccb3b7" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:39.503643 7274 raft_consensus.cc:3032] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 01:59:39.503687 7274 raft_consensus.cc:729] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 01:59:39.503669 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6833c997d284063955d0b9b09ccb3b7" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:39.503818 7014 raft_consensus.cc:3037] T c6833c997d284063955d0b9b09ccb3b7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:39.503849 7274 consensus_queue.cc:260] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:39.504084 7274 raft_consensus.cc:3037] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:39.505089 7274 raft_consensus.cc:2454] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 2.
I20240503 01:59:39.505081 7014 raft_consensus.cc:2454] T c6833c997d284063955d0b9b09ccb3b7 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 2.
I20240503 01:59:39.506316 7079 leader_election.cc:302] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:39.506839 9092 raft_consensus.cc:2781] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 01:59:39.507011 9092 raft_consensus.cc:686] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 2 LEADER]: Becoming Leader. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Running, Role: LEADER
I20240503 01:59:39.507201 9092 consensus_queue.cc:237] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:39.511233 6741 catalog_manager.cc:5484] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: term changed from 1 to 2, leader changed from 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) to c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131). New cstate: current_term: 2 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:39.539119 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6833c997d284063955d0b9b09ccb3b7" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:39.539295 9756 raft_consensus.cc:3037] T c6833c997d284063955d0b9b09ccb3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Advancing to term 2
I20240503 01:59:39.540539 9756 raft_consensus.cc:2454] T c6833c997d284063955d0b9b09ccb3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 2.
I20240503 01:59:39.541963 10668 consensus_queue.cc:1035] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:39.547353 10273 consensus_queue.cc:1035] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:39.607287 6734 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:39.607460 6734 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:39.610621 6734 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 9c589cec02484767aff92eb183f264c4
I20240503 01:59:39.611366 9473 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c589cec02484767aff92eb183f264c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:39 UTC) from {username='slave'} at 127.0.0.1:54678
I20240503 01:59:39.611795 9604 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c589cec02484767aff92eb183f264c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:39 UTC) from {username='slave'} at 127.0.0.1:58562
I20240503 01:59:39.612617 7254 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c589cec02484767aff92eb183f264c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:39 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:39.612787 10924 tablet_replica.cc:330] T 9c589cec02484767aff92eb183f264c4 P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:39.612927 10924 raft_consensus.cc:2229] T 9c589cec02484767aff92eb183f264c4 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.613065 10924 raft_consensus.cc:2258] T 9c589cec02484767aff92eb183f264c4 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.613934 10924 ts_tablet_manager.cc:1875] T 9c589cec02484767aff92eb183f264c4 P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:39.616187 10924 ts_tablet_manager.cc:1888] T 9c589cec02484767aff92eb183f264c4 P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:39.616253 10924 log.cc:1196] T 9c589cec02484767aff92eb183f264c4 P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/9c589cec02484767aff92eb183f264c4
I20240503 01:59:39.616453 10924 ts_tablet_manager.cc:1909] T 9c589cec02484767aff92eb183f264c4 P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:39.617520 7254 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:39.618212 6861 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:39.618305 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet 9c589cec02484767aff92eb183f264c4 (table rebalancer_dynamic_table_new_name [id=6901d9bbd44b491e820471118eab2b38]) successfully deleted
I20240503 01:59:39.623111 6993 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:39.640175 9473 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:39.649655 9604 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:39.661548 9342 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:39.674089 9736 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:39.680740 10078 consensus_queue.cc:1035] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:39.689050 11004 tablet_replica.cc:330] T 9c589cec02484767aff92eb183f264c4 P b7062fbbc2b54fff9e99ec4ea5562c76: stopping tablet replica
I20240503 01:59:39.690148 11003 tablet_replica.cc:330] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a: stopping tablet replica
I20240503 01:59:39.690312 11003 raft_consensus.cc:2229] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:39.690650 11004 raft_consensus.cc:2229] T 9c589cec02484767aff92eb183f264c4 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.690783 11004 raft_consensus.cc:2258] T 9c589cec02484767aff92eb183f264c4 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.691489 11004 ts_tablet_manager.cc:1875] T 9c589cec02484767aff92eb183f264c4 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:39.692557 11003 raft_consensus.cc:2258] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.693213 11003 ts_tablet_manager.cc:1875] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:39.697126 7125 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:39.698736 11004 ts_tablet_manager.cc:1888] T 9c589cec02484767aff92eb183f264c4 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:39.699764 11003 ts_tablet_manager.cc:1888] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:39.699913 11004 log.cc:1196] T 9c589cec02484767aff92eb183f264c4 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/9c589cec02484767aff92eb183f264c4
I20240503 01:59:39.700289 11004 ts_tablet_manager.cc:1909] T 9c589cec02484767aff92eb183f264c4 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting consensus metadata
I20240503 01:59:39.700994 11003 log.cc:1196] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/9c589cec02484767aff92eb183f264c4
I20240503 01:59:39.701205 6727 catalog_manager.cc:4830] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): tablet 9c589cec02484767aff92eb183f264c4 (table rebalancer_dynamic_table_new_name [id=6901d9bbd44b491e820471118eab2b38]) successfully deleted
I20240503 01:59:39.701408 11003 ts_tablet_manager.cc:1909] T 9c589cec02484767aff92eb183f264c4 P bb5364d730cb4489a0bfc6c270b7450a: Deleting consensus metadata
I20240503 01:59:39.702214 6725 catalog_manager.cc:4830] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): tablet 9c589cec02484767aff92eb183f264c4 (table rebalancer_dynamic_table_new_name [id=6901d9bbd44b491e820471118eab2b38]) successfully deleted
I20240503 01:59:39.705925 10078 consensus_queue.cc:1035] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:39.715011 6734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:39.724027 9736 tablet_service.cc:1467] Processing CreateTablet for tablet 4a92c87f064d4a539aed54d8093c7a81 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=aea5812d8261444595ae911598ccaa30]), partition=
I20240503 01:59:39.724316 6993 tablet_service.cc:1467] Processing CreateTablet for tablet 4a92c87f064d4a539aed54d8093c7a81 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=aea5812d8261444595ae911598ccaa30]), partition=
I20240503 01:59:39.724370 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a92c87f064d4a539aed54d8093c7a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:39.724596 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a92c87f064d4a539aed54d8093c7a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:39.727311 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 4a92c87f064d4a539aed54d8093c7a81 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=aea5812d8261444595ae911598ccaa30]), partition=
I20240503 01:59:39.727602 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a92c87f064d4a539aed54d8093c7a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:39.729789 11010 tablet_bootstrap.cc:492] T 4a92c87f064d4a539aed54d8093c7a81 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:39.731089 11010 tablet_bootstrap.cc:654] T 4a92c87f064d4a539aed54d8093c7a81 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:39.732316 11010 tablet_bootstrap.cc:492] T 4a92c87f064d4a539aed54d8093c7a81 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:39.732414 11010 ts_tablet_manager.cc:1367] T 4a92c87f064d4a539aed54d8093c7a81 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:39.732887 11010 raft_consensus.cc:348] T 4a92c87f064d4a539aed54d8093c7a81 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:39.733062 11010 raft_consensus.cc:374] T 4a92c87f064d4a539aed54d8093c7a81 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:39.733129 11010 raft_consensus.cc:729] T 4a92c87f064d4a539aed54d8093c7a81 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:39.733296 11010 consensus_queue.cc:260] T 4a92c87f064d4a539aed54d8093c7a81 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:39.734184 11010 ts_tablet_manager.cc:1398] T 4a92c87f064d4a539aed54d8093c7a81 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:39.738745 11009 tablet_bootstrap.cc:492] T 4a92c87f064d4a539aed54d8093c7a81 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:39.738778 11012 tablet_bootstrap.cc:492] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:39.739854 11009 tablet_bootstrap.cc:654] T 4a92c87f064d4a539aed54d8093c7a81 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:39.739925 11012 tablet_bootstrap.cc:654] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:39.741456 11012 tablet_bootstrap.cc:492] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:39.741611 11012 ts_tablet_manager.cc:1367] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:39.742174 11012 raft_consensus.cc:348] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:39.742322 11012 raft_consensus.cc:374] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:39.742355 11012 raft_consensus.cc:729] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:39.742550 11012 consensus_queue.cc:260] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:39.742916 11012 ts_tablet_manager.cc:1398] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:39.747632 11009 tablet_bootstrap.cc:492] T 4a92c87f064d4a539aed54d8093c7a81 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:39.747776 11009 ts_tablet_manager.cc:1367] T 4a92c87f064d4a539aed54d8093c7a81 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 01:59:39.748328 11009 raft_consensus.cc:348] T 4a92c87f064d4a539aed54d8093c7a81 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:39.748463 11009 raft_consensus.cc:374] T 4a92c87f064d4a539aed54d8093c7a81 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:39.748504 11009 raft_consensus.cc:729] T 4a92c87f064d4a539aed54d8093c7a81 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:39.748672 11009 consensus_queue.cc:260] T 4a92c87f064d4a539aed54d8093c7a81 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:39.748982 11009 ts_tablet_manager.cc:1398] T 4a92c87f064d4a539aed54d8093c7a81 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:39.849844 6735 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240503 01:59:39.850016 6735 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240503 01:59:39.854306 6735 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 4 replicas of tablet 580e6b4d2b2f4f79b8c2af5e75a3c75f
I20240503 01:59:39.855593 7254 tablet_service.cc:1514] Processing DeleteTablet for tablet 580e6b4d2b2f4f79b8c2af5e75a3c75f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:39 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:39.855856 10924 tablet_replica.cc:330] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:39.856034 10924 raft_consensus.cc:2229] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:39.856344 10924 raft_consensus.cc:2258] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.857128 10924 ts_tablet_manager.cc:1875] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:39.860102 10377 consensus_queue.cc:1035] T 5b386bc757e44ca588e3ba6ca76c92b6 P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:39.868371 10924 ts_tablet_manager.cc:1888] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:39.868506 10924 log.cc:1196] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/580e6b4d2b2f4f79b8c2af5e75a3c75f
I20240503 01:59:39.868813 10924 ts_tablet_manager.cc:1909] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:39.870079 6993 tablet_service.cc:1514] Processing DeleteTablet for tablet 580e6b4d2b2f4f79b8c2af5e75a3c75f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:39 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:39.870700 11055 tablet_replica.cc:330] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:39.870707 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet 580e6b4d2b2f4f79b8c2af5e75a3c75f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:39 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:39.870863 11055 raft_consensus.cc:2229] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.871098 11055 raft_consensus.cc:2258] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.871138 11056 tablet_replica.cc:330] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:39.871289 11056 raft_consensus.cc:2229] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.871351 9604 tablet_service.cc:1514] Processing DeleteTablet for tablet 580e6b4d2b2f4f79b8c2af5e75a3c75f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:39 UTC) from {username='slave'} at 127.0.0.1:58562
I20240503 01:59:39.871515 11056 raft_consensus.cc:2258] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.871557 11004 tablet_replica.cc:330] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P b7062fbbc2b54fff9e99ec4ea5562c76: stopping tablet replica
I20240503 01:59:39.871605 6735 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet f8f3bfca68014d818a58484d248a9846
I20240503 01:59:39.871687 11004 raft_consensus.cc:2229] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.871788 11004 raft_consensus.cc:2258] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.871893 11055 ts_tablet_manager.cc:1875] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:39.872040 11056 ts_tablet_manager.cc:1875] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:39.872084 7254 tablet_service.cc:1514] Processing DeleteTablet for tablet f8f3bfca68014d818a58484d248a9846 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:39 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:39.872398 11004 ts_tablet_manager.cc:1875] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:39.872692 6993 tablet_service.cc:1514] Processing DeleteTablet for tablet f8f3bfca68014d818a58484d248a9846 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:39 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:39.873028 6735 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 4 replicas of tablet c35ecf8890b64caaac7e5ea9f6e8f989
I20240503 01:59:39.873420 6993 tablet_service.cc:1514] Processing DeleteTablet for tablet c35ecf8890b64caaac7e5ea9f6e8f989 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:39 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:39.873764 7254 tablet_service.cc:1514] Processing DeleteTablet for tablet c35ecf8890b64caaac7e5ea9f6e8f989 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:39 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:39.874161 6739 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:39.874382 9342 tablet_service.cc:1514] Processing DeleteTablet for tablet c35ecf8890b64caaac7e5ea9f6e8f989 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:39 UTC) from {username='slave'} at 127.0.0.1:36040
I20240503 01:59:39.874733 11004 ts_tablet_manager.cc:1888] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P b7062fbbc2b54fff9e99ec4ea5562c76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:39.874804 11004 log.cc:1196] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/580e6b4d2b2f4f79b8c2af5e75a3c75f
I20240503 01:59:39.874833 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet c35ecf8890b64caaac7e5ea9f6e8f989 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:39 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:39.875095 11004 ts_tablet_manager.cc:1909] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting consensus metadata
I20240503 01:59:39.875319 7125 tablet_service.cc:1514] Processing DeleteTablet for tablet f8f3bfca68014d818a58484d248a9846 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:39 UTC) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:39.875430 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet 580e6b4d2b2f4f79b8c2af5e75a3c75f (table rebalance_test_table_14 [id=c0195ea81aa24df495aff19543a2b8eb]) successfully deleted
I20240503 01:59:39.875648 10923 tablet_replica.cc:330] T f8f3bfca68014d818a58484d248a9846 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:39.875777 10923 raft_consensus.cc:2229] T f8f3bfca68014d818a58484d248a9846 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.875916 10923 raft_consensus.cc:2258] T f8f3bfca68014d818a58484d248a9846 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.876300 11056 ts_tablet_manager.cc:1888] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:39.876372 11056 log.cc:1196] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/580e6b4d2b2f4f79b8c2af5e75a3c75f
I20240503 01:59:39.876425 6727 catalog_manager.cc:4830] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): tablet 580e6b4d2b2f4f79b8c2af5e75a3c75f (table rebalance_test_table_14 [id=c0195ea81aa24df495aff19543a2b8eb]) successfully deleted
I20240503 01:59:39.876505 10923 ts_tablet_manager.cc:1875] T f8f3bfca68014d818a58484d248a9846 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:39.876658 11056 ts_tablet_manager.cc:1909] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:39.877326 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 580e6b4d2b2f4f79b8c2af5e75a3c75f (table rebalance_test_table_14 [id=c0195ea81aa24df495aff19543a2b8eb]) successfully deleted
I20240503 01:59:39.877532 11056 tablet_replica.cc:330] T c35ecf8890b64caaac7e5ea9f6e8f989 P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:39.877646 11056 raft_consensus.cc:2229] T c35ecf8890b64caaac7e5ea9f6e8f989 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.877772 11056 raft_consensus.cc:2258] T c35ecf8890b64caaac7e5ea9f6e8f989 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.878360 11056 ts_tablet_manager.cc:1875] T c35ecf8890b64caaac7e5ea9f6e8f989 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:39.878399 11055 ts_tablet_manager.cc:1888] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:39.878463 11055 log.cc:1196] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/580e6b4d2b2f4f79b8c2af5e75a3c75f
I20240503 01:59:39.878690 11057 tablet_replica.cc:330] T c35ecf8890b64caaac7e5ea9f6e8f989 P 12e8c683db054df3b75c6285b98a5ad8: stopping tablet replica
I20240503 01:59:39.878787 11055 ts_tablet_manager.cc:1909] T 580e6b4d2b2f4f79b8c2af5e75a3c75f P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:39.879101 11057 raft_consensus.cc:2229] T c35ecf8890b64caaac7e5ea9f6e8f989 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.879211 11057 raft_consensus.cc:2258] T c35ecf8890b64caaac7e5ea9f6e8f989 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.879374 11055 tablet_replica.cc:330] T f8f3bfca68014d818a58484d248a9846 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:39.879493 11055 raft_consensus.cc:2229] T f8f3bfca68014d818a58484d248a9846 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.879611 11055 raft_consensus.cc:2258] T f8f3bfca68014d818a58484d248a9846 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.879765 11057 ts_tablet_manager.cc:1875] T c35ecf8890b64caaac7e5ea9f6e8f989 P 12e8c683db054df3b75c6285b98a5ad8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:39.880828 11055 ts_tablet_manager.cc:1875] T f8f3bfca68014d818a58484d248a9846 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:39.882823 11055 ts_tablet_manager.cc:1888] T f8f3bfca68014d818a58484d248a9846 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:39.882882 11055 log.cc:1196] T f8f3bfca68014d818a58484d248a9846 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/f8f3bfca68014d818a58484d248a9846
I20240503 01:59:39.883143 11055 ts_tablet_manager.cc:1909] T f8f3bfca68014d818a58484d248a9846 P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:39.883750 11055 tablet_replica.cc:330] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:39.883869 11055 raft_consensus.cc:2229] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:39.884050 11055 raft_consensus.cc:2258] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.884735 11055 ts_tablet_manager.cc:1875] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:39.885952 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet 580e6b4d2b2f4f79b8c2af5e75a3c75f (table rebalance_test_table_14 [id=c0195ea81aa24df495aff19543a2b8eb]) successfully deleted
I20240503 01:59:39.886102 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet f8f3bfca68014d818a58484d248a9846 (table rebalance_test_table_14 [id=c0195ea81aa24df495aff19543a2b8eb]) successfully deleted
I20240503 01:59:39.887691 11055 ts_tablet_manager.cc:1888] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:39.887794 11055 log.cc:1196] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/c35ecf8890b64caaac7e5ea9f6e8f989
I20240503 01:59:39.888432 9605 tablet_service.cc:1467] Processing CreateTablet for tablet 2f271b67a6f34eb18c98824ca4b26d17 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=5094b51906a7434daee33202abd81bfc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:39.888574 9473 tablet_service.cc:1467] Processing CreateTablet for tablet 31f5289f1dff4fcaa3492c16e40a3ed7 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=5094b51906a7434daee33202abd81bfc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:39.888610 9474 tablet_service.cc:1467] Processing CreateTablet for tablet 1b6f98b01ead4cea8f362dc283caa901 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=5094b51906a7434daee33202abd81bfc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:39.888733 9605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f271b67a6f34eb18c98824ca4b26d17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:39.888566 11055 ts_tablet_manager.cc:1909] T c35ecf8890b64caaac7e5ea9f6e8f989 P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:39.888844 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f5289f1dff4fcaa3492c16e40a3ed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:39.889465 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet c35ecf8890b64caaac7e5ea9f6e8f989 (table rebalance_test_table_14 [id=c0195ea81aa24df495aff19543a2b8eb]) successfully deleted
I20240503 01:59:39.891170 9604 tablet_service.cc:1467] Processing CreateTablet for tablet 31f5289f1dff4fcaa3492c16e40a3ed7 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=5094b51906a7434daee33202abd81bfc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:39.891422 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f5289f1dff4fcaa3492c16e40a3ed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:39.891685 9736 tablet_service.cc:1467] Processing CreateTablet for tablet 31f5289f1dff4fcaa3492c16e40a3ed7 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=5094b51906a7434daee33202abd81bfc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:39.891717 9735 tablet_service.cc:1467] Processing CreateTablet for tablet 1b6f98b01ead4cea8f362dc283caa901 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=5094b51906a7434daee33202abd81bfc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:39.891924 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f5289f1dff4fcaa3492c16e40a3ed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:39.892091 9735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b6f98b01ead4cea8f362dc283caa901. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:39.892323 9737 tablet_service.cc:1467] Processing CreateTablet for tablet 2f271b67a6f34eb18c98824ca4b26d17 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=5094b51906a7434daee33202abd81bfc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:39.892561 9737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f271b67a6f34eb18c98824ca4b26d17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:39.892989 7125 tablet_service.cc:1467] Processing CreateTablet for tablet 2f271b67a6f34eb18c98824ca4b26d17 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=5094b51906a7434daee33202abd81bfc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:39.893234 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f271b67a6f34eb18c98824ca4b26d17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:39.894363 11009 tablet_bootstrap.cc:492] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:39.895406 11009 tablet_bootstrap.cc:654] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:39.896018 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 1b6f98b01ead4cea8f362dc283caa901 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=5094b51906a7434daee33202abd81bfc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:39.896293 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b6f98b01ead4cea8f362dc283caa901. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:39.897550 9474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b6f98b01ead4cea8f362dc283caa901. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:39.897867 10377 consensus_queue.cc:1035] T 5b386bc757e44ca588e3ba6ca76c92b6 P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:39.898672 11010 tablet_bootstrap.cc:492] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:39.899569 11010 tablet_bootstrap.cc:654] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:39.901899 11010 tablet_bootstrap.cc:492] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:39.901988 11010 ts_tablet_manager.cc:1367] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:59:39.902467 11010 raft_consensus.cc:348] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:39.902617 11010 raft_consensus.cc:374] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:39.902655 11010 raft_consensus.cc:729] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:39.902737 11058 tablet_bootstrap.cc:492] T 2f271b67a6f34eb18c98824ca4b26d17 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:39.902774 11010 consensus_queue.cc:260] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:39.903077 11010 ts_tablet_manager.cc:1398] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:39.903241 10923 ts_tablet_manager.cc:1888] T f8f3bfca68014d818a58484d248a9846 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:39.903504 10923 log.cc:1196] T f8f3bfca68014d818a58484d248a9846 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/f8f3bfca68014d818a58484d248a9846
I20240503 01:59:39.903771 11058 tablet_bootstrap.cc:654] T 2f271b67a6f34eb18c98824ca4b26d17 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:39.903851 10923 ts_tablet_manager.cc:1909] T f8f3bfca68014d818a58484d248a9846 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting consensus metadata
I20240503 01:59:39.904615 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet f8f3bfca68014d818a58484d248a9846 (table rebalance_test_table_14 [id=c0195ea81aa24df495aff19543a2b8eb]) successfully deleted
I20240503 01:59:39.906272 10924 tablet_replica.cc:330] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:39.906512 10924 raft_consensus.cc:2229] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:39.906829 11058 tablet_bootstrap.cc:492] T 2f271b67a6f34eb18c98824ca4b26d17 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:39.906850 10924 raft_consensus.cc:2258] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.906973 11058 ts_tablet_manager.cc:1367] T 2f271b67a6f34eb18c98824ca4b26d17 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:39.907441 10924 ts_tablet_manager.cc:1875] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:39.907441 11058 raft_consensus.cc:348] T 2f271b67a6f34eb18c98824ca4b26d17 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:39.907634 11058 raft_consensus.cc:374] T 2f271b67a6f34eb18c98824ca4b26d17 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:39.907703 11058 raft_consensus.cc:729] T 2f271b67a6f34eb18c98824ca4b26d17 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:39.907876 11058 consensus_queue.cc:260] T 2f271b67a6f34eb18c98824ca4b26d17 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:39.907951 10079 raft_consensus.cc:482] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:39.908066 10079 raft_consensus.cc:504] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:39.908447 10079 leader_election.cc:288] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:39.908785 9362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a92c87f064d4a539aed54d8093c7a81" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:39.908936 9362 raft_consensus.cc:2454] T 4a92c87f064d4a539aed54d8093c7a81 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:39.908954 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a92c87f064d4a539aed54d8093c7a81" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:39.909098 9757 raft_consensus.cc:2454] T 4a92c87f064d4a539aed54d8093c7a81 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:39.909281 6949 leader_election.cc:302] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:39.909510 10079 raft_consensus.cc:2781] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:39.909559 10079 raft_consensus.cc:482] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:39.909591 10079 raft_consensus.cc:3037] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:39.910717 10079 raft_consensus.cc:504] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:39.911180 11058 ts_tablet_manager.cc:1398] T 2f271b67a6f34eb18c98824ca4b26d17 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:39.911366 10079 leader_election.cc:288] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:39.911617 11058 tablet_bootstrap.cc:492] T 31f5289f1dff4fcaa3492c16e40a3ed7 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:39.912925 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a92c87f064d4a539aed54d8093c7a81" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:39.913116 9757 raft_consensus.cc:3037] T 4a92c87f064d4a539aed54d8093c7a81 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:39.913215 11056 ts_tablet_manager.cc:1888] T c35ecf8890b64caaac7e5ea9f6e8f989 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:39.910238 10924 ts_tablet_manager.cc:1888] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:39.914237 10924 log.cc:1196] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/f8f3bfca68014d818a58484d248a9846
I20240503 01:59:39.914319 9757 raft_consensus.cc:2454] T 4a92c87f064d4a539aed54d8093c7a81 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:39.914664 10924 ts_tablet_manager.cc:1909] T f8f3bfca68014d818a58484d248a9846 P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:39.914713 11058 tablet_bootstrap.cc:654] T 31f5289f1dff4fcaa3492c16e40a3ed7 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:39.915659 6948 leader_election.cc:302] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:39.915807 9755 raft_consensus.cc:1261] T c6833c997d284063955d0b9b09ccb3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 2 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 01:59:39.916280 9362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a92c87f064d4a539aed54d8093c7a81" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:39.916400 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet f8f3bfca68014d818a58484d248a9846 (table rebalance_test_table_14 [id=c0195ea81aa24df495aff19543a2b8eb]) successfully deleted
I20240503 01:59:39.916478 9362 raft_consensus.cc:3037] T 4a92c87f064d4a539aed54d8093c7a81 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:39.916476 10953 consensus_queue.cc:1035] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:39.916857 10924 tablet_replica.cc:330] T c35ecf8890b64caaac7e5ea9f6e8f989 P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:39.917086 10924 raft_consensus.cc:2229] T c35ecf8890b64caaac7e5ea9f6e8f989 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:39.917877 11059 tablet_bootstrap.cc:492] T 2f271b67a6f34eb18c98824ca4b26d17 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:39.918089 9362 raft_consensus.cc:2454] T 4a92c87f064d4a539aed54d8093c7a81 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:39.918159 11061 tablet_bootstrap.cc:492] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:39.919117 10079 raft_consensus.cc:2781] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:39.919222 10079 raft_consensus.cc:686] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:39.919364 10079 consensus_queue.cc:237] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:39.919517 11061 tablet_bootstrap.cc:654] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:39.920020 11056 log.cc:1196] T c35ecf8890b64caaac7e5ea9f6e8f989 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/c35ecf8890b64caaac7e5ea9f6e8f989
I20240503 01:59:39.920344 10924 raft_consensus.cc:2258] T c35ecf8890b64caaac7e5ea9f6e8f989 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:39.921376 10924 ts_tablet_manager.cc:1875] T c35ecf8890b64caaac7e5ea9f6e8f989 P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:39.924877 11057 ts_tablet_manager.cc:1888] T c35ecf8890b64caaac7e5ea9f6e8f989 P 12e8c683db054df3b75c6285b98a5ad8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:39.924959 11057 log.cc:1196] T c35ecf8890b64caaac7e5ea9f6e8f989 P 12e8c683db054df3b75c6285b98a5ad8: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/c35ecf8890b64caaac7e5ea9f6e8f989
I20240503 01:59:39.925215 11057 ts_tablet_manager.cc:1909] T c35ecf8890b64caaac7e5ea9f6e8f989 P 12e8c683db054df3b75c6285b98a5ad8: Deleting consensus metadata
I20240503 01:59:39.925954 11056 ts_tablet_manager.cc:1909] T c35ecf8890b64caaac7e5ea9f6e8f989 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:39.926425 11009 tablet_bootstrap.cc:492] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:39.926548 11009 ts_tablet_manager.cc:1367] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.000s
I20240503 01:59:39.925853 6735 catalog_manager.cc:5484] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:39.926848 11058 tablet_bootstrap.cc:492] T 31f5289f1dff4fcaa3492c16e40a3ed7 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:39.926934 11058 ts_tablet_manager.cc:1367] T 31f5289f1dff4fcaa3492c16e40a3ed7 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 01:59:39.927284 11009 raft_consensus.cc:348] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:39.927383 11009 raft_consensus.cc:374] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:39.927410 11009 raft_consensus.cc:729] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:39.927379 11058 raft_consensus.cc:348] T 31f5289f1dff4fcaa3492c16e40a3ed7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:39.927492 11058 raft_consensus.cc:374] T 31f5289f1dff4fcaa3492c16e40a3ed7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:39.927525 11058 raft_consensus.cc:729] T 31f5289f1dff4fcaa3492c16e40a3ed7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:39.927518 11009 consensus_queue.cc:260] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:39.927649 11058 consensus_queue.cc:260] T 31f5289f1dff4fcaa3492c16e40a3ed7 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:39.927794 11009 ts_tablet_manager.cc:1398] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:39.927950 11009 tablet_bootstrap.cc:492] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:39.928419 11058 ts_tablet_manager.cc:1398] T 31f5289f1dff4fcaa3492c16e40a3ed7 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:39.928895 11009 tablet_bootstrap.cc:654] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:39.929186 6727 catalog_manager.cc:4830] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): tablet c35ecf8890b64caaac7e5ea9f6e8f989 (table rebalance_test_table_14 [id=c0195ea81aa24df495aff19543a2b8eb]) successfully deleted
I20240503 01:59:39.932376 11059 tablet_bootstrap.cc:654] T 2f271b67a6f34eb18c98824ca4b26d17 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:39.933789 10078 raft_consensus.cc:482] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:39.933887 10078 raft_consensus.cc:504] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:39.934265 10078 leader_election.cc:288] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:39.935123 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6f98b01ead4cea8f362dc283caa901" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
W20240503 01:59:39.935887 9296 leader_election.cc:341] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Illegal state: must be running to vote when last-logged opid is not known
I20240503 01:59:39.936529 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet c35ecf8890b64caaac7e5ea9f6e8f989 (table rebalance_test_table_14 [id=c0195ea81aa24df495aff19543a2b8eb]) successfully deleted
I20240503 01:59:39.941579 10924 ts_tablet_manager.cc:1888] T c35ecf8890b64caaac7e5ea9f6e8f989 P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:39.941814 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6f98b01ead4cea8f362dc283caa901" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:39.941960 9757 raft_consensus.cc:2454] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:39.942256 9297 leader_election.cc:302] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, db1a0000aaa346f79c0aa4b6c5105191; no voters: bb5364d730cb4489a0bfc6c270b7450a
I20240503 01:59:39.942549 10078 raft_consensus.cc:2781] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:39.942597 10078 raft_consensus.cc:482] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:39.942631 10078 raft_consensus.cc:3037] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:39.943580 10078 raft_consensus.cc:504] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:39.943938 10078 leader_election.cc:288] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Requested vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:39.944371 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6f98b01ead4cea8f362dc283caa901" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:39.944494 9755 raft_consensus.cc:3037] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:39.944849 11061 tablet_bootstrap.cc:492] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:39.944977 11061 ts_tablet_manager.cc:1367] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20240503 01:59:39.945407 9755 raft_consensus.cc:2454] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:39.945732 9297 leader_election.cc:302] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:39.945901 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6f98b01ead4cea8f362dc283caa901" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:39.946004 10078 raft_consensus.cc:2781] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:39.946098 10078 raft_consensus.cc:686] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Running, Role: LEADER
I20240503 01:59:39.946259 10078 consensus_queue.cc:237] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
W20240503 01:59:39.946353 9296 leader_election.cc:341] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Illegal state: must be running to vote when last-logged opid is not known
I20240503 01:59:39.947023 11061 raft_consensus.cc:348] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:39.947222 11061 raft_consensus.cc:374] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:39.947310 11061 raft_consensus.cc:729] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:39.947532 11061 consensus_queue.cc:260] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:39.950119 6738 catalog_manager.cc:5484] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:39.955137 11061 ts_tablet_manager.cc:1398] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 01:59:39.955157 11009 tablet_bootstrap.cc:492] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:39.955277 11009 ts_tablet_manager.cc:1367] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20240503 01:59:39.955511 11061 tablet_bootstrap.cc:492] T 1b6f98b01ead4cea8f362dc283caa901 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:39.956110 11009 raft_consensus.cc:348] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:39.956224 11009 raft_consensus.cc:374] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:39.956256 11009 raft_consensus.cc:729] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:39.956882 11061 tablet_bootstrap.cc:654] T 1b6f98b01ead4cea8f362dc283caa901 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:39.957017 11009 consensus_queue.cc:260] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:39.957298 11009 ts_tablet_manager.cc:1398] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:39.957484 11009 tablet_bootstrap.cc:492] T 31f5289f1dff4fcaa3492c16e40a3ed7 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:39.958451 11009 tablet_bootstrap.cc:654] T 31f5289f1dff4fcaa3492c16e40a3ed7 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:39.960440 10924 log.cc:1196] T c35ecf8890b64caaac7e5ea9f6e8f989 P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/c35ecf8890b64caaac7e5ea9f6e8f989
I20240503 01:59:39.960994 10924 ts_tablet_manager.cc:1909] T c35ecf8890b64caaac7e5ea9f6e8f989 P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:39.962018 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet c35ecf8890b64caaac7e5ea9f6e8f989 (table rebalance_test_table_14 [id=c0195ea81aa24df495aff19543a2b8eb]) successfully deleted
I20240503 01:59:39.964566 7275 raft_consensus.cc:1261] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [term 2 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 01:59:39.964502 11059 tablet_bootstrap.cc:492] T 2f271b67a6f34eb18c98824ca4b26d17 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:39.966141 11059 ts_tablet_manager.cc:1367] T 2f271b67a6f34eb18c98824ca4b26d17 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.000s
I20240503 01:59:39.968253 11059 raft_consensus.cc:348] T 2f271b67a6f34eb18c98824ca4b26d17 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:39.968552 11059 raft_consensus.cc:374] T 2f271b67a6f34eb18c98824ca4b26d17 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:39.968649 11059 raft_consensus.cc:729] T 2f271b67a6f34eb18c98824ca4b26d17 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:39.965095 9092 consensus_queue.cc:1035] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 01:59:39.979638 7125 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:39.983036 11059 consensus_queue.cc:260] T 2f271b67a6f34eb18c98824ca4b26d17 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:39.983980 9604 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:39.986101 6993 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:39.990778 6861 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:39.991214 9736 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:39.991628 7254 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:39.998993 11059 ts_tablet_manager.cc:1398] T 2f271b67a6f34eb18c98824ca4b26d17 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.031s user 0.001s sys 0.000s
I20240503 01:59:40.006997 9342 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:39.979997 9474 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 01:59:40.019678 11061 tablet_bootstrap.cc:492] T 1b6f98b01ead4cea8f362dc283caa901 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:40.019873 11061 ts_tablet_manager.cc:1367] T 1b6f98b01ead4cea8f362dc283caa901 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.000s
I20240503 01:59:40.020411 11061 raft_consensus.cc:348] T 1b6f98b01ead4cea8f362dc283caa901 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:40.020545 11061 raft_consensus.cc:374] T 1b6f98b01ead4cea8f362dc283caa901 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:40.020576 11061 raft_consensus.cc:729] T 1b6f98b01ead4cea8f362dc283caa901 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:40.020726 11061 consensus_queue.cc:260] T 1b6f98b01ead4cea8f362dc283caa901 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:40.021451 11061 ts_tablet_manager.cc:1398] T 1b6f98b01ead4cea8f362dc283caa901 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:40.025133 7012 raft_consensus.cc:1261] T c6833c997d284063955d0b9b09ccb3b7 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 01:59:40.025552 9092 consensus_queue.cc:1035] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.038815 6741 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:40.048112 11009 tablet_bootstrap.cc:492] T 31f5289f1dff4fcaa3492c16e40a3ed7 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:40.048246 11009 ts_tablet_manager.cc:1367] T 31f5289f1dff4fcaa3492c16e40a3ed7 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.091s user 0.002s sys 0.000s
I20240503 01:59:40.048802 11009 raft_consensus.cc:348] T 31f5289f1dff4fcaa3492c16e40a3ed7 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:40.049008 11009 raft_consensus.cc:374] T 31f5289f1dff4fcaa3492c16e40a3ed7 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:40.049088 11009 raft_consensus.cc:729] T 31f5289f1dff4fcaa3492c16e40a3ed7 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:40.049263 11009 consensus_queue.cc:260] T 31f5289f1dff4fcaa3492c16e40a3ed7 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:40.050113 11009 ts_tablet_manager.cc:1398] T 31f5289f1dff4fcaa3492c16e40a3ed7 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:40.062762 9755 raft_consensus.cc:1261] T 4a92c87f064d4a539aed54d8093c7a81 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:40.062927 9362 raft_consensus.cc:1261] T 4a92c87f064d4a539aed54d8093c7a81 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:40.063386 10855 consensus_queue.cc:1035] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:40.063566 10079 consensus_queue.cc:1035] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:40.066139 6741 catalog_manager.cc:5576] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053) does not have the latest schema for tablet 4a92c87f064d4a539aed54d8093c7a81 (table rebalancer_dynamic_table [id=aea5812d8261444595ae911598ccaa30]). Expected version 1 got 0
I20240503 01:59:40.067411 6738 catalog_manager.cc:5576] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387) does not have the latest schema for tablet 4a92c87f064d4a539aed54d8093c7a81 (table rebalancer_dynamic_table [id=aea5812d8261444595ae911598ccaa30]). Expected version 1 got 0
I20240503 01:59:40.078441 11094 tablet.cc:1722] T 4a92c87f064d4a539aed54d8093c7a81 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:40.079329 11094 tablet.cc:1620] T 4a92c87f064d4a539aed54d8093c7a81 P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:40.079840 6738 catalog_manager.cc:5576] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053) does not have the latest schema for tablet 4a92c87f064d4a539aed54d8093c7a81 (table rebalancer_dynamic_table [id=aea5812d8261444595ae911598ccaa30]). Expected version 1 got 0
I20240503 01:59:40.082665 11095 tablet.cc:1722] T 4a92c87f064d4a539aed54d8093c7a81 P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:40.082751 6738 catalog_manager.cc:5576] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387) does not have the latest schema for tablet 4a92c87f064d4a539aed54d8093c7a81 (table rebalancer_dynamic_table [id=aea5812d8261444595ae911598ccaa30]). Expected version 1 got 0
I20240503 01:59:40.083093 11095 tablet.cc:1620] T 4a92c87f064d4a539aed54d8093c7a81 P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 01:59:40.091611 6738 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=aea5812d8261444595ae911598ccaa30] alter complete (version 1)
I20240503 01:59:40.092325 11093 tablet.cc:1722] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:40.092653 11093 tablet.cc:1620] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef: MemRowSet was empty: no flush needed.
I20240503 01:59:40.127277 7275 consensus_queue.cc:237] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:40.128644 9363 raft_consensus.cc:1261] T a5e23eebc629494cbb8262bc123a720a P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.128705 7012 raft_consensus.cc:1261] T a5e23eebc629494cbb8262bc123a720a P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.129099 10389 consensus_queue.cc:1035] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.129282 10585 consensus_queue.cc:1035] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.131052 11114 raft_consensus.cc:2932] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:40.131410 9363 raft_consensus.cc:2932] T a5e23eebc629494cbb8262bc123a720a P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:40.131434 7012 raft_consensus.cc:2932] T a5e23eebc629494cbb8262bc123a720a P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:40.134197 6735 catalog_manager.cc:5484] T a5e23eebc629494cbb8262bc123a720a P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:40.137604 7125 tablet_service.cc:1514] Processing DeleteTablet for tablet a5e23eebc629494cbb8262bc123a720a with delete_type TABLET_DATA_TOMBSTONED (TS c688f3df56ac4b6d9b0c30fa4cc8ca2a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:40.137862 10923 tablet_replica.cc:330] T a5e23eebc629494cbb8262bc123a720a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:40.138015 10923 raft_consensus.cc:2229] T a5e23eebc629494cbb8262bc123a720a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.138165 10923 raft_consensus.cc:2258] T a5e23eebc629494cbb8262bc123a720a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.138638 10923 ts_tablet_manager.cc:1875] T a5e23eebc629494cbb8262bc123a720a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:40.140291 10547 raft_consensus.cc:482] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:40.140413 10547 raft_consensus.cc:504] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:40.140862 10547 leader_election.cc:288] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:40.140992 10923 ts_tablet_manager.cc:1888] T a5e23eebc629494cbb8262bc123a720a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 01:59:40.141065 10923 log.cc:1196] T a5e23eebc629494cbb8262bc123a720a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/a5e23eebc629494cbb8262bc123a720a
I20240503 01:59:40.141644 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet a5e23eebc629494cbb8262bc123a720a (table rebalance_test_table_3 [id=45e94c90bda847688db4572a4f9d0c02]) successfully deleted
I20240503 01:59:40.142112 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f271b67a6f34eb18c98824ca4b26d17" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:40.142277 9625 raft_consensus.cc:2454] T 2f271b67a6f34eb18c98824ca4b26d17 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 01:59:40.142714 9692 leader_election.cc:302] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:40.142890 10547 raft_consensus.cc:2781] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:40.142938 10547 raft_consensus.cc:482] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:40.142976 10547 raft_consensus.cc:3037] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:40.143791 10547 raft_consensus.cc:504] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:40.144106 10547 leader_election.cc:288] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Requested vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:40.144554 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f271b67a6f34eb18c98824ca4b26d17" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:40.144691 9625 raft_consensus.cc:3037] T 2f271b67a6f34eb18c98824ca4b26d17 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:40.145695 9625 raft_consensus.cc:2454] T 2f271b67a6f34eb18c98824ca4b26d17 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:40.146023 9692 leader_election.cc:302] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:40.146194 10547 raft_consensus.cc:2781] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:40.146378 10547 raft_consensus.cc:686] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Becoming Leader. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Running, Role: LEADER
I20240503 01:59:40.146550 10547 consensus_queue.cc:237] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:40.148036 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f271b67a6f34eb18c98824ca4b26d17" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:40.148064 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f271b67a6f34eb18c98824ca4b26d17" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:40.148182 7143 raft_consensus.cc:2454] T 2f271b67a6f34eb18c98824ca4b26d17 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 01:59:40.148368 6738 catalog_manager.cc:5484] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: term changed from 0 to 1, leader changed from <none> to db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:40.162891 10359 raft_consensus.cc:482] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:40.163017 10359 raft_consensus.cc:504] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:40.163417 10359 leader_election.cc:288] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:40.163808 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f5289f1dff4fcaa3492c16e40a3ed7" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:40.163846 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f5289f1dff4fcaa3492c16e40a3ed7" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:40.163944 9755 raft_consensus.cc:2454] T 31f5289f1dff4fcaa3492c16e40a3ed7 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:40.164009 9625 raft_consensus.cc:2454] T 31f5289f1dff4fcaa3492c16e40a3ed7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:40.164247 9428 leader_election.cc:302] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:40.164394 10359 raft_consensus.cc:2781] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:40.164454 10359 raft_consensus.cc:482] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:40.164494 10359 raft_consensus.cc:3037] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:40.165377 10359 raft_consensus.cc:504] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:40.165699 10359 leader_election.cc:288] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Requested vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:40.165974 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f5289f1dff4fcaa3492c16e40a3ed7" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:40.166030 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f5289f1dff4fcaa3492c16e40a3ed7" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:40.166085 9755 raft_consensus.cc:3037] T 31f5289f1dff4fcaa3492c16e40a3ed7 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:40.166126 9625 raft_consensus.cc:3037] T 31f5289f1dff4fcaa3492c16e40a3ed7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:40.167054 9755 raft_consensus.cc:2454] T 31f5289f1dff4fcaa3492c16e40a3ed7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:40.167054 9625 raft_consensus.cc:2454] T 31f5289f1dff4fcaa3492c16e40a3ed7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:40.167388 9429 leader_election.cc:302] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:40.167791 10359 raft_consensus.cc:2781] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:40.167891 10359 raft_consensus.cc:686] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Becoming Leader. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Running, Role: LEADER
I20240503 01:59:40.168017 10359 consensus_queue.cc:237] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:40.169817 6738 catalog_manager.cc:5484] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: term changed from 0 to 1, leader changed from <none> to bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:40.200302 6740 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:40.205618 11093 tablet.cc:1722] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:40.205996 11093 tablet.cc:1620] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef: MemRowSet was empty: no flush needed.
I20240503 01:59:40.205991 11095 tablet.cc:1722] T 4a92c87f064d4a539aed54d8093c7a81 P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:40.206301 11095 tablet.cc:1620] T 4a92c87f064d4a539aed54d8093c7a81 P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 01:59:40.209153 11094 tablet.cc:1722] T 4a92c87f064d4a539aed54d8093c7a81 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:40.209455 11094 tablet.cc:1620] T 4a92c87f064d4a539aed54d8093c7a81 P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:40.212159 6740 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=aea5812d8261444595ae911598ccaa30] alter complete (version 2)
I20240503 01:59:40.228170 9342 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:40.234310 6993 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:40.236838 9474 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:40.237365 9604 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:40.238406 7254 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:40.239969 7125 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:40.257601 6861 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:40.269176 9736 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:40.317834 6740 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:40.322913 11093 tablet.cc:1722] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:40.323714 11095 tablet.cc:1722] T 4a92c87f064d4a539aed54d8093c7a81 P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:40.324041 11094 tablet.cc:1722] T 4a92c87f064d4a539aed54d8093c7a81 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:40.326607 10078 consensus_queue.cc:1035] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:40.327409 6882 consensus_queue.cc:237] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:40.329669 9757 raft_consensus.cc:1261] T dec7e3036d384f5db51f50ba93acbbc6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.329823 7275 raft_consensus.cc:1261] T dec7e3036d384f5db51f50ba93acbbc6 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.330024 10668 consensus_queue.cc:1035] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.330350 10668 consensus_queue.cc:1035] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.331624 6738 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=aea5812d8261444595ae911598ccaa30] alter complete (version 3)
I20240503 01:59:40.334329 10273 raft_consensus.cc:2932] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:40.336623 7275 raft_consensus.cc:2932] T dec7e3036d384f5db51f50ba93acbbc6 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:40.338129 6740 catalog_manager.cc:5484] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:40.341172 7125 tablet_service.cc:1514] Processing DeleteTablet for tablet dec7e3036d384f5db51f50ba93acbbc6 with delete_type TABLET_DATA_TOMBSTONED (TS c688f3df56ac4b6d9b0c30fa4cc8ca2a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:40.341388 10923 tablet_replica.cc:330] T dec7e3036d384f5db51f50ba93acbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:40.341536 10923 raft_consensus.cc:2229] T dec7e3036d384f5db51f50ba93acbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.341696 10923 raft_consensus.cc:2258] T dec7e3036d384f5db51f50ba93acbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.342159 10923 ts_tablet_manager.cc:1875] T dec7e3036d384f5db51f50ba93acbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:40.344367 10923 ts_tablet_manager.cc:1888] T dec7e3036d384f5db51f50ba93acbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 01:59:40.344444 10923 log.cc:1196] T dec7e3036d384f5db51f50ba93acbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/dec7e3036d384f5db51f50ba93acbbc6
I20240503 01:59:40.345606 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet dec7e3036d384f5db51f50ba93acbbc6 (table rebalance_test_table_17 [id=b061a61957e74150a4af26c16c277330]) successfully deleted
I20240503 01:59:40.345728 9494 raft_consensus.cc:3037] T 1b6f98b01ead4cea8f362dc283caa901 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:40.346897 9757 raft_consensus.cc:2932] T dec7e3036d384f5db51f50ba93acbbc6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:40.351487 11193 consensus_queue.cc:1035] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:40.377517 6740 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240503 01:59:40.377678 6740 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240503 01:59:40.380964 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet b14a9a0bd39843cf8132e067fc48c29e
I20240503 01:59:40.381351 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 96c8dabb170f44fd98bd467f62255713
I20240503 01:59:40.381611 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 424123bb8be2460b8855b5a9ad785e57
I20240503 01:59:40.381637 7125 tablet_service.cc:1514] Processing DeleteTablet for tablet b14a9a0bd39843cf8132e067fc48c29e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:40 UTC) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:40.381657 6993 tablet_service.cc:1514] Processing DeleteTablet for tablet b14a9a0bd39843cf8132e067fc48c29e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:40 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:40.381748 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet b14a9a0bd39843cf8132e067fc48c29e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:40 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:40.381844 10923 tablet_replica.cc:330] T b14a9a0bd39843cf8132e067fc48c29e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:40.381916 11056 tablet_replica.cc:330] T b14a9a0bd39843cf8132e067fc48c29e P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:40.381980 10923 raft_consensus.cc:2229] T b14a9a0bd39843cf8132e067fc48c29e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.382022 11055 tablet_replica.cc:330] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:40.382035 11056 raft_consensus.cc:2229] T b14a9a0bd39843cf8132e067fc48c29e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.382118 10923 raft_consensus.cc:2258] T b14a9a0bd39843cf8132e067fc48c29e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.382153 11056 raft_consensus.cc:2258] T b14a9a0bd39843cf8132e067fc48c29e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.382158 11055 raft_consensus.cc:2229] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:40.382375 11055 raft_consensus.cc:2258] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.382480 7254 tablet_service.cc:1514] Processing DeleteTablet for tablet 96c8dabb170f44fd98bd467f62255713 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:40 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:40.382648 11056 ts_tablet_manager.cc:1875] T b14a9a0bd39843cf8132e067fc48c29e P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:40.382665 10923 ts_tablet_manager.cc:1875] T b14a9a0bd39843cf8132e067fc48c29e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:40.382987 11055 ts_tablet_manager.cc:1875] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:40.383236 7254 tablet_service.cc:1514] Processing DeleteTablet for tablet 424123bb8be2460b8855b5a9ad785e57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:40 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:40.384027 7125 tablet_service.cc:1514] Processing DeleteTablet for tablet 424123bb8be2460b8855b5a9ad785e57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:40 UTC) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:40.384807 11056 ts_tablet_manager.cc:1888] T b14a9a0bd39843cf8132e067fc48c29e P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:40.384807 10923 ts_tablet_manager.cc:1888] T b14a9a0bd39843cf8132e067fc48c29e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:40.384882 11056 log.cc:1196] T b14a9a0bd39843cf8132e067fc48c29e P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/b14a9a0bd39843cf8132e067fc48c29e
I20240503 01:59:40.384894 10923 log.cc:1196] T b14a9a0bd39843cf8132e067fc48c29e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/b14a9a0bd39843cf8132e067fc48c29e
I20240503 01:59:40.385165 10923 ts_tablet_manager.cc:1909] T b14a9a0bd39843cf8132e067fc48c29e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting consensus metadata
I20240503 01:59:40.385165 11056 ts_tablet_manager.cc:1909] T b14a9a0bd39843cf8132e067fc48c29e P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:40.385272 11055 ts_tablet_manager.cc:1888] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:40.385331 11055 log.cc:1196] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/b14a9a0bd39843cf8132e067fc48c29e
I20240503 01:59:40.385385 10924 tablet_replica.cc:330] T 96c8dabb170f44fd98bd467f62255713 P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:40.385522 10924 raft_consensus.cc:2229] T 96c8dabb170f44fd98bd467f62255713 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.385607 11055 ts_tablet_manager.cc:1909] T b14a9a0bd39843cf8132e067fc48c29e P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:40.385686 10924 raft_consensus.cc:2258] T 96c8dabb170f44fd98bd467f62255713 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.386068 6994 tablet_service.cc:1514] Processing DeleteTablet for tablet 424123bb8be2460b8855b5a9ad785e57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:40 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:40.386132 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet b14a9a0bd39843cf8132e067fc48c29e (table rebalance_test_table_16 [id=f40118346c7d461fb1d445e8dab8b379]) successfully deleted
I20240503 01:59:40.386190 6993 tablet_service.cc:1514] Processing DeleteTablet for tablet 96c8dabb170f44fd98bd467f62255713 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:40 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:40.386221 10924 ts_tablet_manager.cc:1875] T 96c8dabb170f44fd98bd467f62255713 P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:40.386353 10923 tablet_replica.cc:330] T 424123bb8be2460b8855b5a9ad785e57 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:40.386485 10923 raft_consensus.cc:2229] T 424123bb8be2460b8855b5a9ad785e57 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.386857 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet b14a9a0bd39843cf8132e067fc48c29e (table rebalance_test_table_16 [id=f40118346c7d461fb1d445e8dab8b379]) successfully deleted
I20240503 01:59:40.387262 11055 tablet_replica.cc:330] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:40.387444 11055 raft_consensus.cc:2229] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:40.387686 11055 raft_consensus.cc:2258] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.387881 10923 raft_consensus.cc:2258] T 424123bb8be2460b8855b5a9ad785e57 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.388077 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet b14a9a0bd39843cf8132e067fc48c29e (table rebalance_test_table_16 [id=f40118346c7d461fb1d445e8dab8b379]) successfully deleted
I20240503 01:59:40.388274 11055 ts_tablet_manager.cc:1875] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:40.388374 10924 ts_tablet_manager.cc:1888] T 96c8dabb170f44fd98bd467f62255713 P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:40.388445 10924 log.cc:1196] T 96c8dabb170f44fd98bd467f62255713 P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/96c8dabb170f44fd98bd467f62255713
I20240503 01:59:40.388516 10923 ts_tablet_manager.cc:1875] T 424123bb8be2460b8855b5a9ad785e57 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:40.388700 10924 ts_tablet_manager.cc:1909] T 96c8dabb170f44fd98bd467f62255713 P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:40.388890 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet 96c8dabb170f44fd98bd467f62255713 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:40 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:40.389139 11056 tablet_replica.cc:330] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:40.389298 11056 raft_consensus.cc:2229] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:40.389536 11056 raft_consensus.cc:2258] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.390050 11056 ts_tablet_manager.cc:1875] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:40.390592 10923 ts_tablet_manager.cc:1888] T 424123bb8be2460b8855b5a9ad785e57 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:40.391181 10924 tablet_replica.cc:330] T 424123bb8be2460b8855b5a9ad785e57 P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:40.391372 10924 raft_consensus.cc:2229] T 424123bb8be2460b8855b5a9ad785e57 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.391585 10924 raft_consensus.cc:2258] T 424123bb8be2460b8855b5a9ad785e57 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.391587 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet 96c8dabb170f44fd98bd467f62255713 (table rebalance_test_table_16 [id=f40118346c7d461fb1d445e8dab8b379]) successfully deleted
I20240503 01:59:40.391779 11055 ts_tablet_manager.cc:1888] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:40.391930 11055 log.cc:1196] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/424123bb8be2460b8855b5a9ad785e57
I20240503 01:59:40.392346 10924 ts_tablet_manager.cc:1875] T 424123bb8be2460b8855b5a9ad785e57 P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:40.392423 11055 ts_tablet_manager.cc:1909] T 424123bb8be2460b8855b5a9ad785e57 P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:40.392479 11056 ts_tablet_manager.cc:1888] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:40.392604 11056 log.cc:1196] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/96c8dabb170f44fd98bd467f62255713
I20240503 01:59:40.393175 11056 ts_tablet_manager.cc:1909] T 96c8dabb170f44fd98bd467f62255713 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:40.393560 11055 tablet_replica.cc:330] T 96c8dabb170f44fd98bd467f62255713 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:40.393743 11055 raft_consensus.cc:2229] T 96c8dabb170f44fd98bd467f62255713 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.393939 11055 raft_consensus.cc:2258] T 96c8dabb170f44fd98bd467f62255713 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.394063 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet 424123bb8be2460b8855b5a9ad785e57 (table rebalance_test_table_16 [id=f40118346c7d461fb1d445e8dab8b379]) successfully deleted
I20240503 01:59:40.394299 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 96c8dabb170f44fd98bd467f62255713 (table rebalance_test_table_16 [id=f40118346c7d461fb1d445e8dab8b379]) successfully deleted
I20240503 01:59:40.394529 11055 ts_tablet_manager.cc:1875] T 96c8dabb170f44fd98bd467f62255713 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:40.394589 10924 ts_tablet_manager.cc:1888] T 424123bb8be2460b8855b5a9ad785e57 P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:40.394609 10923 log.cc:1196] T 424123bb8be2460b8855b5a9ad785e57 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/424123bb8be2460b8855b5a9ad785e57
I20240503 01:59:40.394734 10924 log.cc:1196] T 424123bb8be2460b8855b5a9ad785e57 P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/424123bb8be2460b8855b5a9ad785e57
I20240503 01:59:40.394896 10923 ts_tablet_manager.cc:1909] T 424123bb8be2460b8855b5a9ad785e57 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting consensus metadata
I20240503 01:59:40.395087 10924 ts_tablet_manager.cc:1909] T 424123bb8be2460b8855b5a9ad785e57 P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:40.395643 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet 424123bb8be2460b8855b5a9ad785e57 (table rebalance_test_table_16 [id=f40118346c7d461fb1d445e8dab8b379]) successfully deleted
I20240503 01:59:40.396168 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet 424123bb8be2460b8855b5a9ad785e57 (table rebalance_test_table_16 [id=f40118346c7d461fb1d445e8dab8b379]) successfully deleted
I20240503 01:59:40.397410 11055 ts_tablet_manager.cc:1888] T 96c8dabb170f44fd98bd467f62255713 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:40.397564 11055 log.cc:1196] T 96c8dabb170f44fd98bd467f62255713 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/96c8dabb170f44fd98bd467f62255713
I20240503 01:59:40.397914 11055 ts_tablet_manager.cc:1909] T 96c8dabb170f44fd98bd467f62255713 P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:40.399648 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet 96c8dabb170f44fd98bd467f62255713 (table rebalance_test_table_16 [id=f40118346c7d461fb1d445e8dab8b379]) successfully deleted
I20240503 01:59:40.435072 7254 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:40.435979 9604 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:40.436053 9342 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:40.438107 6733 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:40.438252 6733 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:40.442628 9474 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:40.447435 6733 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 4a92c87f064d4a539aed54d8093c7a81
I20240503 01:59:40.448590 6993 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a92c87f064d4a539aed54d8093c7a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:40 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:40.448943 9736 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a92c87f064d4a539aed54d8093c7a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:40 UTC) from {username='slave'} at 127.0.0.1:48152
I20240503 01:59:40.448952 9342 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a92c87f064d4a539aed54d8093c7a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:40 UTC) from {username='slave'} at 127.0.0.1:36040
I20240503 01:59:40.449206 11055 tablet_replica.cc:330] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:40.449358 11055 raft_consensus.cc:2229] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:40.449576 11055 raft_consensus.cc:2258] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.449671 6861 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:40.450350 11055 ts_tablet_manager.cc:1875] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:40.452718 11055 ts_tablet_manager.cc:1888] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:40.452800 11055 log.cc:1196] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/4a92c87f064d4a539aed54d8093c7a81
I20240503 01:59:40.453130 11055 ts_tablet_manager.cc:1909] T 4a92c87f064d4a539aed54d8093c7a81 P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:40.456936 7125 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:40.457505 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet 4a92c87f064d4a539aed54d8093c7a81 (table rebalancer_dynamic_table_new_name [id=aea5812d8261444595ae911598ccaa30]) successfully deleted
I20240503 01:59:40.461163 11241 tablet_replica.cc:330] T 4a92c87f064d4a539aed54d8093c7a81 P db1a0000aaa346f79c0aa4b6c5105191: stopping tablet replica
I20240503 01:59:40.461413 11241 raft_consensus.cc:2229] T 4a92c87f064d4a539aed54d8093c7a81 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.462960 6993 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:40.458949 11242 tablet_replica.cc:330] T 4a92c87f064d4a539aed54d8093c7a81 P 12e8c683db054df3b75c6285b98a5ad8: stopping tablet replica
I20240503 01:59:40.467196 11242 raft_consensus.cc:2229] T 4a92c87f064d4a539aed54d8093c7a81 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.467424 11242 raft_consensus.cc:2258] T 4a92c87f064d4a539aed54d8093c7a81 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.468171 11242 ts_tablet_manager.cc:1875] T 4a92c87f064d4a539aed54d8093c7a81 P 12e8c683db054df3b75c6285b98a5ad8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:40.471406 11242 ts_tablet_manager.cc:1888] T 4a92c87f064d4a539aed54d8093c7a81 P 12e8c683db054df3b75c6285b98a5ad8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:40.471581 11242 log.cc:1196] T 4a92c87f064d4a539aed54d8093c7a81 P 12e8c683db054df3b75c6285b98a5ad8: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/4a92c87f064d4a539aed54d8093c7a81
I20240503 01:59:40.471899 11242 ts_tablet_manager.cc:1909] T 4a92c87f064d4a539aed54d8093c7a81 P 12e8c683db054df3b75c6285b98a5ad8: Deleting consensus metadata
I20240503 01:59:40.472676 6727 catalog_manager.cc:4830] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): tablet 4a92c87f064d4a539aed54d8093c7a81 (table rebalancer_dynamic_table_new_name [id=aea5812d8261444595ae911598ccaa30]) successfully deleted
I20240503 01:59:40.486713 9736 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:40.492350 11241 raft_consensus.cc:2258] T 4a92c87f064d4a539aed54d8093c7a81 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.493199 11241 ts_tablet_manager.cc:1875] T 4a92c87f064d4a539aed54d8093c7a81 P db1a0000aaa346f79c0aa4b6c5105191: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:40.495174 11241 ts_tablet_manager.cc:1888] T 4a92c87f064d4a539aed54d8093c7a81 P db1a0000aaa346f79c0aa4b6c5105191: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:40.495334 11241 log.cc:1196] T 4a92c87f064d4a539aed54d8093c7a81 P db1a0000aaa346f79c0aa4b6c5105191: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/4a92c87f064d4a539aed54d8093c7a81
I20240503 01:59:40.495671 11241 ts_tablet_manager.cc:1909] T 4a92c87f064d4a539aed54d8093c7a81 P db1a0000aaa346f79c0aa4b6c5105191: Deleting consensus metadata
I20240503 01:59:40.497036 6726 catalog_manager.cc:4830] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): tablet 4a92c87f064d4a539aed54d8093c7a81 (table rebalancer_dynamic_table_new_name [id=aea5812d8261444595ae911598ccaa30]) successfully deleted
I20240503 01:59:40.530498 7275 consensus_queue.cc:237] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:40.531944 9363 raft_consensus.cc:1261] T 6d4480c8de204f5397dbdcff91db97f6 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.531946 7143 raft_consensus.cc:1261] T 6d4480c8de204f5397dbdcff91db97f6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.532335 11114 consensus_queue.cc:1035] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.532512 10585 consensus_queue.cc:1035] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.538960 10585 raft_consensus.cc:2932] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:40.539502 9363 raft_consensus.cc:2932] T 6d4480c8de204f5397dbdcff91db97f6 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:40.540463 7145 raft_consensus.cc:2932] T 6d4480c8de204f5397dbdcff91db97f6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:40.542356 6733 catalog_manager.cc:5484] T 6d4480c8de204f5397dbdcff91db97f6 P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: config changed from index 4 to 5, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:40.545378 6993 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d4480c8de204f5397dbdcff91db97f6 with delete_type TABLET_DATA_TOMBSTONED (TS 63331f455be049c1bdd6c23c59ac1bef not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:40.545626 11055 tablet_replica.cc:330] T 6d4480c8de204f5397dbdcff91db97f6 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:40.545797 11055 raft_consensus.cc:2229] T 6d4480c8de204f5397dbdcff91db97f6 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.545943 11055 raft_consensus.cc:2258] T 6d4480c8de204f5397dbdcff91db97f6 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.546448 11055 ts_tablet_manager.cc:1875] T 6d4480c8de204f5397dbdcff91db97f6 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:40.549048 11055 ts_tablet_manager.cc:1888] T 6d4480c8de204f5397dbdcff91db97f6 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 01:59:40.549121 11055 log.cc:1196] T 6d4480c8de204f5397dbdcff91db97f6 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/6d4480c8de204f5397dbdcff91db97f6
I20240503 01:59:40.549614 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet 6d4480c8de204f5397dbdcff91db97f6 (table rebalance_test_table_5 [id=eca57da7c1764bd9a55819692641dac8]) successfully deleted
I20240503 01:59:40.550825 6737 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:40.556805 9604 tablet_service.cc:1467] Processing CreateTablet for tablet 83366a94af3f47caaafd57be6293c138 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1848b5ab1d7143aa833d7913acd549fe]), partition=
I20240503 01:59:40.557060 9736 tablet_service.cc:1467] Processing CreateTablet for tablet 83366a94af3f47caaafd57be6293c138 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1848b5ab1d7143aa833d7913acd549fe]), partition=
I20240503 01:59:40.557102 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83366a94af3f47caaafd57be6293c138. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:40.557273 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83366a94af3f47caaafd57be6293c138. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:40.558722 6993 tablet_service.cc:1467] Processing CreateTablet for tablet 83366a94af3f47caaafd57be6293c138 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1848b5ab1d7143aa833d7913acd549fe]), partition=
I20240503 01:59:40.558962 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83366a94af3f47caaafd57be6293c138. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:40.561537 11273 tablet_bootstrap.cc:492] T 83366a94af3f47caaafd57be6293c138 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:40.563027 11273 tablet_bootstrap.cc:654] T 83366a94af3f47caaafd57be6293c138 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:40.563639 11275 tablet_bootstrap.cc:492] T 83366a94af3f47caaafd57be6293c138 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:40.563812 11274 tablet_bootstrap.cc:492] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:40.564491 11275 tablet_bootstrap.cc:654] T 83366a94af3f47caaafd57be6293c138 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:40.564716 11274 tablet_bootstrap.cc:654] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:40.565836 11273 tablet_bootstrap.cc:492] T 83366a94af3f47caaafd57be6293c138 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:40.565939 11273 ts_tablet_manager.cc:1367] T 83366a94af3f47caaafd57be6293c138 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:40.566439 11273 raft_consensus.cc:348] T 83366a94af3f47caaafd57be6293c138 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:40.566612 11273 raft_consensus.cc:374] T 83366a94af3f47caaafd57be6293c138 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:40.566677 11273 raft_consensus.cc:729] T 83366a94af3f47caaafd57be6293c138 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:40.566855 11273 consensus_queue.cc:260] T 83366a94af3f47caaafd57be6293c138 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:40.567308 11273 ts_tablet_manager.cc:1398] T 83366a94af3f47caaafd57be6293c138 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:40.567584 11275 tablet_bootstrap.cc:492] T 83366a94af3f47caaafd57be6293c138 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:40.567687 11275 ts_tablet_manager.cc:1367] T 83366a94af3f47caaafd57be6293c138 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:40.567782 11274 tablet_bootstrap.cc:492] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:40.567876 11274 ts_tablet_manager.cc:1367] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:40.568219 11275 raft_consensus.cc:348] T 83366a94af3f47caaafd57be6293c138 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:40.568392 11275 raft_consensus.cc:374] T 83366a94af3f47caaafd57be6293c138 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:40.568382 11274 raft_consensus.cc:348] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:40.568468 11275 raft_consensus.cc:729] T 83366a94af3f47caaafd57be6293c138 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:40.568547 11274 raft_consensus.cc:374] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:40.568624 11274 raft_consensus.cc:729] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:40.568645 11275 consensus_queue.cc:260] T 83366a94af3f47caaafd57be6293c138 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:40.568902 11274 consensus_queue.cc:260] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:40.569715 11274 ts_tablet_manager.cc:1398] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:40.569768 11275 ts_tablet_manager.cc:1398] T 83366a94af3f47caaafd57be6293c138 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:40.575181 10547 raft_consensus.cc:482] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:40.575299 10547 raft_consensus.cc:504] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:40.575789 10547 leader_election.cc:288] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:40.576129 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83366a94af3f47caaafd57be6293c138" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:40.576265 9625 raft_consensus.cc:2454] T 83366a94af3f47caaafd57be6293c138 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 01:59:40.576263 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83366a94af3f47caaafd57be6293c138" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:40.576400 7012 raft_consensus.cc:2454] T 83366a94af3f47caaafd57be6293c138 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 01:59:40.576722 9692 leader_election.cc:302] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:40.576906 10547 raft_consensus.cc:2781] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:40.576962 10547 raft_consensus.cc:482] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:40.576997 10547 raft_consensus.cc:3037] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:40.577982 10547 raft_consensus.cc:504] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:40.578265 10547 leader_election.cc:288] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:40.578696 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83366a94af3f47caaafd57be6293c138" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:40.578830 9625 raft_consensus.cc:3037] T 83366a94af3f47caaafd57be6293c138 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:40.579775 9625 raft_consensus.cc:2454] T 83366a94af3f47caaafd57be6293c138 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:40.580241 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83366a94af3f47caaafd57be6293c138" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:40.580360 7012 raft_consensus.cc:3037] T 83366a94af3f47caaafd57be6293c138 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:40.581269 7012 raft_consensus.cc:2454] T 83366a94af3f47caaafd57be6293c138 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:40.581550 9690 leader_election.cc:302] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:40.581807 10547 raft_consensus.cc:2781] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:40.581882 10547 raft_consensus.cc:686] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Becoming Leader. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Running, Role: LEADER
I20240503 01:59:40.582019 10547 consensus_queue.cc:237] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:40.583855 6733 catalog_manager.cc:5484] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: term changed from 0 to 1, leader changed from <none> to db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:40.631803 10547 consensus_queue.cc:1035] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:40.631927 10359 consensus_queue.cc:1035] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:40.636132 9342 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:40.637883 7254 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:40.643841 10869 consensus_queue.cc:1035] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:40.649230 6861 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:40.650444 9736 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:40.658335 9604 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:40.658336 6993 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:40.672451 9474 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 01:59:40.680158 7145 raft_consensus.cc:3037] T 2f271b67a6f34eb18c98824ca4b26d17 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:40.681708 11127 consensus_queue.cc:1035] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:40.683816 7125 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:40.690277 6732 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:40.694083 9626 raft_consensus.cc:1261] T 83366a94af3f47caaafd57be6293c138 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:40.694085 7012 raft_consensus.cc:1261] T 83366a94af3f47caaafd57be6293c138 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:40.694495 11127 consensus_queue.cc:1035] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:40.694850 10547 consensus_queue.cc:1035] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:40.697149 6732 catalog_manager.cc:5576] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231) does not have the latest schema for tablet 83366a94af3f47caaafd57be6293c138 (table rebalancer_dynamic_table [id=1848b5ab1d7143aa833d7913acd549fe]). Expected version 1 got 0
I20240503 01:59:40.699726 11094 tablet.cc:1722] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:40.700102 11094 tablet.cc:1620] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:40.700697 11093 tablet.cc:1722] T 83366a94af3f47caaafd57be6293c138 P 63331f455be049c1bdd6c23c59ac1bef: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:40.700986 11093 tablet.cc:1620] T 83366a94af3f47caaafd57be6293c138 P 63331f455be049c1bdd6c23c59ac1bef: MemRowSet was empty: no flush needed.
I20240503 01:59:40.703791 6732 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:40.704317 6738 catalog_manager.cc:5576] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087) does not have the latest schema for tablet 83366a94af3f47caaafd57be6293c138 (table rebalancer_dynamic_table [id=1848b5ab1d7143aa833d7913acd549fe]). Expected version 1 got 0
I20240503 01:59:40.706574 6741 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=1848b5ab1d7143aa833d7913acd549fe] alter complete (version 1)
I20240503 01:59:40.709812 6738 catalog_manager.cc:5576] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087) does not have the latest schema for tablet 83366a94af3f47caaafd57be6293c138 (table rebalancer_dynamic_table [id=1848b5ab1d7143aa833d7913acd549fe]). Expected version 1 got 0
I20240503 01:59:40.711118 11327 tablet.cc:1722] T 83366a94af3f47caaafd57be6293c138 P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:40.711453 11327 tablet.cc:1620] T 83366a94af3f47caaafd57be6293c138 P b7062fbbc2b54fff9e99ec4ea5562c76: MemRowSet was empty: no flush needed.
I20240503 01:59:40.713219 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 84812ece4f27484eb008bfbebdbfee84 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=15afeaf3f27142ebaf9ef93cac9e4dd6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:40.713249 9343 tablet_service.cc:1467] Processing CreateTablet for tablet 3e824fcc9aac449d9373655b304010e2 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=15afeaf3f27142ebaf9ef93cac9e4dd6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:40.713253 7125 tablet_service.cc:1467] Processing CreateTablet for tablet 84812ece4f27484eb008bfbebdbfee84 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=15afeaf3f27142ebaf9ef93cac9e4dd6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:40.713514 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84812ece4f27484eb008bfbebdbfee84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:40.713505 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84812ece4f27484eb008bfbebdbfee84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:40.713896 9604 tablet_service.cc:1467] Processing CreateTablet for tablet 43b252f688d74d9b833dcd976b74149b (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=15afeaf3f27142ebaf9ef93cac9e4dd6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:40.714121 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43b252f688d74d9b833dcd976b74149b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:40.714578 6993 tablet_service.cc:1467] Processing CreateTablet for tablet 43b252f688d74d9b833dcd976b74149b (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=15afeaf3f27142ebaf9ef93cac9e4dd6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:40.714838 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43b252f688d74d9b833dcd976b74149b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:40.715955 9736 tablet_service.cc:1467] Processing CreateTablet for tablet 84812ece4f27484eb008bfbebdbfee84 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=15afeaf3f27142ebaf9ef93cac9e4dd6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:40.716215 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84812ece4f27484eb008bfbebdbfee84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:40.716331 7124 tablet_service.cc:1467] Processing CreateTablet for tablet 3e824fcc9aac449d9373655b304010e2 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=15afeaf3f27142ebaf9ef93cac9e4dd6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:40.716568 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e824fcc9aac449d9373655b304010e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:40.716611 9343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e824fcc9aac449d9373655b304010e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:40.716990 11275 tablet_bootstrap.cc:492] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:40.717916 11275 tablet_bootstrap.cc:654] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:40.718667 11273 tablet_bootstrap.cc:492] T 43b252f688d74d9b833dcd976b74149b P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:40.719256 9474 tablet_service.cc:1467] Processing CreateTablet for tablet 3e824fcc9aac449d9373655b304010e2 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=15afeaf3f27142ebaf9ef93cac9e4dd6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:40.719537 11273 tablet_bootstrap.cc:654] T 43b252f688d74d9b833dcd976b74149b P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:40.719549 9474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e824fcc9aac449d9373655b304010e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:40.720135 11274 tablet_bootstrap.cc:492] T 84812ece4f27484eb008bfbebdbfee84 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:40.720557 11275 tablet_bootstrap.cc:492] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:40.720640 11275 ts_tablet_manager.cc:1367] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:40.721047 11328 tablet_bootstrap.cc:492] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:40.721066 11274 tablet_bootstrap.cc:654] T 84812ece4f27484eb008bfbebdbfee84 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:40.721112 11275 raft_consensus.cc:348] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:40.721231 11275 raft_consensus.cc:374] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:40.721263 11275 raft_consensus.cc:729] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:40.721379 11275 consensus_queue.cc:260] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:40.721661 11275 ts_tablet_manager.cc:1398] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:40.721804 11333 tablet_bootstrap.cc:492] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:40.721887 11328 tablet_bootstrap.cc:654] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:40.721961 9473 tablet_service.cc:1467] Processing CreateTablet for tablet 43b252f688d74d9b833dcd976b74149b (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=15afeaf3f27142ebaf9ef93cac9e4dd6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:40.722172 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43b252f688d74d9b833dcd976b74149b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:40.722812 11333 tablet_bootstrap.cc:654] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:40.723156 11328 tablet_bootstrap.cc:492] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:40.723234 11328 ts_tablet_manager.cc:1367] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:40.723340 11273 tablet_bootstrap.cc:492] T 43b252f688d74d9b833dcd976b74149b P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:40.723412 11273 ts_tablet_manager.cc:1367] T 43b252f688d74d9b833dcd976b74149b P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240503 01:59:40.723685 11328 raft_consensus.cc:348] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:40.723781 11328 raft_consensus.cc:374] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:40.723819 11328 raft_consensus.cc:729] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:40.723824 11274 tablet_bootstrap.cc:492] T 84812ece4f27484eb008bfbebdbfee84 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:40.723826 11273 raft_consensus.cc:348] T 43b252f688d74d9b833dcd976b74149b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:40.723912 11274 ts_tablet_manager.cc:1367] T 84812ece4f27484eb008bfbebdbfee84 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:40.723922 11273 raft_consensus.cc:374] T 43b252f688d74d9b833dcd976b74149b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:40.723953 11273 raft_consensus.cc:729] T 43b252f688d74d9b833dcd976b74149b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:40.723948 11328 consensus_queue.cc:260] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:40.724061 11273 consensus_queue.cc:260] T 43b252f688d74d9b833dcd976b74149b P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:40.724220 11328 ts_tablet_manager.cc:1398] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:40.724321 11273 ts_tablet_manager.cc:1398] T 43b252f688d74d9b833dcd976b74149b P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:40.724386 11328 tablet_bootstrap.cc:492] T 3e824fcc9aac449d9373655b304010e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:40.724376 11274 raft_consensus.cc:348] T 84812ece4f27484eb008bfbebdbfee84 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:40.724476 11274 raft_consensus.cc:374] T 84812ece4f27484eb008bfbebdbfee84 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:40.724517 11274 raft_consensus.cc:729] T 84812ece4f27484eb008bfbebdbfee84 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:40.724634 11274 consensus_queue.cc:260] T 84812ece4f27484eb008bfbebdbfee84 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:40.724884 11274 ts_tablet_manager.cc:1398] T 84812ece4f27484eb008bfbebdbfee84 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:40.725216 11328 tablet_bootstrap.cc:654] T 3e824fcc9aac449d9373655b304010e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:40.726565 11328 tablet_bootstrap.cc:492] T 3e824fcc9aac449d9373655b304010e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:40.726642 11328 ts_tablet_manager.cc:1367] T 3e824fcc9aac449d9373655b304010e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:40.727077 11328 raft_consensus.cc:348] T 3e824fcc9aac449d9373655b304010e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:40.727182 11328 raft_consensus.cc:374] T 3e824fcc9aac449d9373655b304010e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:40.727212 11328 raft_consensus.cc:729] T 3e824fcc9aac449d9373655b304010e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:40.727329 11328 consensus_queue.cc:260] T 3e824fcc9aac449d9373655b304010e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:40.728091 11328 ts_tablet_manager.cc:1398] T 3e824fcc9aac449d9373655b304010e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:40.735500 11329 tablet_bootstrap.cc:492] T 84812ece4f27484eb008bfbebdbfee84 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:40.736657 11329 tablet_bootstrap.cc:654] T 84812ece4f27484eb008bfbebdbfee84 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:40.737759 11329 tablet_bootstrap.cc:492] T 84812ece4f27484eb008bfbebdbfee84 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:40.737762 11333 tablet_bootstrap.cc:492] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:40.737854 11333 ts_tablet_manager.cc:1367] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 01:59:40.737906 11329 ts_tablet_manager.cc:1367] T 84812ece4f27484eb008bfbebdbfee84 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:40.738338 11329 raft_consensus.cc:348] T 84812ece4f27484eb008bfbebdbfee84 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:40.738338 11333 raft_consensus.cc:348] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:40.738461 11329 raft_consensus.cc:374] T 84812ece4f27484eb008bfbebdbfee84 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:40.738461 11333 raft_consensus.cc:374] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:40.738554 11329 raft_consensus.cc:729] T 84812ece4f27484eb008bfbebdbfee84 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:40.738502 11333 raft_consensus.cc:729] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:40.738718 11329 consensus_queue.cc:260] T 84812ece4f27484eb008bfbebdbfee84 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:40.738725 11333 consensus_queue.cc:260] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:40.739100 11329 ts_tablet_manager.cc:1398] T 84812ece4f27484eb008bfbebdbfee84 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:40.739097 11333 ts_tablet_manager.cc:1398] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:40.739264 11333 tablet_bootstrap.cc:492] T 43b252f688d74d9b833dcd976b74149b P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:40.739293 11329 tablet_bootstrap.cc:492] T 3e824fcc9aac449d9373655b304010e2 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:40.741919 11333 tablet_bootstrap.cc:654] T 43b252f688d74d9b833dcd976b74149b P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:40.742398 11329 tablet_bootstrap.cc:654] T 3e824fcc9aac449d9373655b304010e2 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:40.743397 11329 tablet_bootstrap.cc:492] T 3e824fcc9aac449d9373655b304010e2 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:40.743489 11329 ts_tablet_manager.cc:1367] T 3e824fcc9aac449d9373655b304010e2 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:40.743854 11329 raft_consensus.cc:348] T 3e824fcc9aac449d9373655b304010e2 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:40.743947 11329 raft_consensus.cc:374] T 3e824fcc9aac449d9373655b304010e2 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:40.743983 11329 raft_consensus.cc:729] T 3e824fcc9aac449d9373655b304010e2 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:40.744128 11329 consensus_queue.cc:260] T 3e824fcc9aac449d9373655b304010e2 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:40.744421 11329 ts_tablet_manager.cc:1398] T 3e824fcc9aac449d9373655b304010e2 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:40.745505 11333 tablet_bootstrap.cc:492] T 43b252f688d74d9b833dcd976b74149b P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:40.745615 11333 ts_tablet_manager.cc:1367] T 43b252f688d74d9b833dcd976b74149b P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:40.746055 11333 raft_consensus.cc:348] T 43b252f688d74d9b833dcd976b74149b P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:40.746205 11333 raft_consensus.cc:374] T 43b252f688d74d9b833dcd976b74149b P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:40.746248 11333 raft_consensus.cc:729] T 43b252f688d74d9b833dcd976b74149b P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:40.746385 11333 consensus_queue.cc:260] T 43b252f688d74d9b833dcd976b74149b P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:40.746830 11333 ts_tablet_manager.cc:1398] T 43b252f688d74d9b833dcd976b74149b P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:40.765584 7275 consensus_queue.cc:237] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:40.766973 6882 raft_consensus.cc:1261] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.767000 9363 raft_consensus.cc:1261] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.767397 11114 consensus_queue.cc:1035] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.767570 10585 consensus_queue.cc:1035] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.767902 9092 raft_consensus.cc:482] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:40.767997 9092 raft_consensus.cc:504] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:40.768344 9092 leader_election.cc:288] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:40.768919 9362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84812ece4f27484eb008bfbebdbfee84" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:40.768971 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84812ece4f27484eb008bfbebdbfee84" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:40.769045 9362 raft_consensus.cc:2454] T 84812ece4f27484eb008bfbebdbfee84 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:40.769080 9757 raft_consensus.cc:2454] T 84812ece4f27484eb008bfbebdbfee84 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:40.769352 7079 leader_election.cc:302] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:40.769412 11114 raft_consensus.cc:2932] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:40.769531 9092 raft_consensus.cc:2781] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:40.769579 9092 raft_consensus.cc:482] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:40.769623 9092 raft_consensus.cc:3037] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:40.770395 9092 raft_consensus.cc:504] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:40.770782 9092 leader_election.cc:288] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Requested vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:40.771203 6882 raft_consensus.cc:2932] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:40.771888 9362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84812ece4f27484eb008bfbebdbfee84" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:40.771996 9362 raft_consensus.cc:3037] T 84812ece4f27484eb008bfbebdbfee84 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:40.773005 9362 raft_consensus.cc:2454] T 84812ece4f27484eb008bfbebdbfee84 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:40.774192 6733 catalog_manager.cc:5484] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:40.776070 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84812ece4f27484eb008bfbebdbfee84" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:40.776212 9757 raft_consensus.cc:3037] T 84812ece4f27484eb008bfbebdbfee84 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:40.777031 9757 raft_consensus.cc:2454] T 84812ece4f27484eb008bfbebdbfee84 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:40.777334 7079 leader_election.cc:302] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:40.778966 9092 raft_consensus.cc:2781] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:40.779070 9092 raft_consensus.cc:686] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Becoming Leader. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Running, Role: LEADER
I20240503 01:59:40.779222 9092 consensus_queue.cc:237] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:40.781358 9363 raft_consensus.cc:2932] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:40.785001 7124 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c7396d1b98c4b7ca495ba3ab5dcbbc6 with delete_type TABLET_DATA_TOMBSTONED (TS c688f3df56ac4b6d9b0c30fa4cc8ca2a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:40.785208 10923 tablet_replica.cc:330] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:40.785334 10923 raft_consensus.cc:2229] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.785466 10923 raft_consensus.cc:2258] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.785509 6741 catalog_manager.cc:5484] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:40.786012 10923 ts_tablet_manager.cc:1875] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:40.788282 10923 ts_tablet_manager.cc:1888] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 01:59:40.788362 10923 log.cc:1196] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/9c7396d1b98c4b7ca495ba3ab5dcbbc6
I20240503 01:59:40.788870 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet 9c7396d1b98c4b7ca495ba3ab5dcbbc6 (table rebalance_test_table_15 [id=76d0e687959345af9b8f30fe2c35d640]) successfully deleted
I20240503 01:59:40.803074 10855 raft_consensus.cc:482] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:40.803210 10855 raft_consensus.cc:504] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:40.803783 10855 leader_election.cc:288] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:40.804093 9626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43b252f688d74d9b833dcd976b74149b" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:40.804235 9626 raft_consensus.cc:2454] T 43b252f688d74d9b833dcd976b74149b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:40.804879 6949 leader_election.cc:302] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:40.805060 10855 raft_consensus.cc:2781] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:40.805123 10855 raft_consensus.cc:482] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:40.805176 10855 raft_consensus.cc:3037] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:40.806216 10855 raft_consensus.cc:504] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:40.806653 10855 leader_election.cc:288] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:40.806864 9626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43b252f688d74d9b833dcd976b74149b" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:40.806975 9626 raft_consensus.cc:3037] T 43b252f688d74d9b833dcd976b74149b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:40.807888 9626 raft_consensus.cc:2454] T 43b252f688d74d9b833dcd976b74149b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:40.808461 6949 leader_election.cc:302] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:40.808622 10855 raft_consensus.cc:2781] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:40.808764 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43b252f688d74d9b833dcd976b74149b" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:40.808908 9493 raft_consensus.cc:3037] T 43b252f688d74d9b833dcd976b74149b P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:40.808948 10855 raft_consensus.cc:686] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:40.809098 10855 consensus_queue.cc:237] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:40.809819 9493 raft_consensus.cc:2454] T 43b252f688d74d9b833dcd976b74149b P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:40.808764 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43b252f688d74d9b833dcd976b74149b" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:40.810297 9494 raft_consensus.cc:2362] T 43b252f688d74d9b833dcd976b74149b P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1. Re-sending same reply.
I20240503 01:59:40.811223 6733 catalog_manager.cc:5484] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:40.812486 6741 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:40.817550 11094 tablet.cc:1722] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:40.817886 11094 tablet.cc:1620] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:40.817906 11093 tablet.cc:1722] T 83366a94af3f47caaafd57be6293c138 P 63331f455be049c1bdd6c23c59ac1bef: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:40.817982 11327 tablet.cc:1722] T 83366a94af3f47caaafd57be6293c138 P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:40.818181 11093 tablet.cc:1620] T 83366a94af3f47caaafd57be6293c138 P 63331f455be049c1bdd6c23c59ac1bef: MemRowSet was empty: no flush needed.
I20240503 01:59:40.818305 11327 tablet.cc:1620] T 83366a94af3f47caaafd57be6293c138 P b7062fbbc2b54fff9e99ec4ea5562c76: MemRowSet was empty: no flush needed.
I20240503 01:59:40.825510 6741 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=1848b5ab1d7143aa833d7913acd549fe] alter complete (version 2)
I20240503 01:59:40.843310 10869 raft_consensus.cc:482] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:40.843466 10869 raft_consensus.cc:504] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:40.844028 10869 leader_election.cc:288] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:40.844381 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e824fcc9aac449d9373655b304010e2" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:40.844522 7145 raft_consensus.cc:2454] T 3e824fcc9aac449d9373655b304010e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:40.844758 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e824fcc9aac449d9373655b304010e2" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:40.844810 9429 leader_election.cc:302] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:40.844894 9363 raft_consensus.cc:2454] T 3e824fcc9aac449d9373655b304010e2 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:40.845026 10869 raft_consensus.cc:2781] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:40.845068 10869 raft_consensus.cc:482] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:40.845103 10869 raft_consensus.cc:3037] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:40.846362 10869 raft_consensus.cc:504] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:40.846737 10869 leader_election.cc:288] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Requested vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:40.847864 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e824fcc9aac449d9373655b304010e2" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:40.847990 9363 raft_consensus.cc:3037] T 3e824fcc9aac449d9373655b304010e2 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:40.849112 9363 raft_consensus.cc:2454] T 3e824fcc9aac449d9373655b304010e2 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:40.849733 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e824fcc9aac449d9373655b304010e2" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:40.849862 7145 raft_consensus.cc:3037] T 3e824fcc9aac449d9373655b304010e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:40.850993 7145 raft_consensus.cc:2454] T 3e824fcc9aac449d9373655b304010e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:40.851796 9429 leader_election.cc:302] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:40.852646 10869 raft_consensus.cc:2781] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:40.852737 10869 raft_consensus.cc:686] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Becoming Leader. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Running, Role: LEADER
I20240503 01:59:40.852870 10869 consensus_queue.cc:237] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:40.856390 6738 catalog_manager.cc:5484] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: term changed from 0 to 1, leader changed from <none> to bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:40.876341 7254 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:40.878638 9343 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:40.879194 6993 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:40.879349 9604 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:40.880031 6861 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:40.891382 6737 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240503 01:59:40.891554 6737 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240503 01:59:40.894037 6737 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 902a8727fa65497aafbc703f3101126b
I20240503 01:59:40.894354 6737 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 556a177350b9483c861972c842a8d193
I20240503 01:59:40.894583 6737 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 249513672b7043d6adcf95ac5ac10a73
I20240503 01:59:40.894857 7124 tablet_service.cc:1514] Processing DeleteTablet for tablet 556a177350b9483c861972c842a8d193 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:40 UTC) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:40.895100 10923 tablet_replica.cc:330] T 556a177350b9483c861972c842a8d193 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:40.895254 10923 raft_consensus.cc:2229] T 556a177350b9483c861972c842a8d193 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.895440 10923 raft_consensus.cc:2258] T 556a177350b9483c861972c842a8d193 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.895990 10923 ts_tablet_manager.cc:1875] T 556a177350b9483c861972c842a8d193 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:40.896382 6994 tablet_service.cc:1514] Processing DeleteTablet for tablet 556a177350b9483c861972c842a8d193 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:40 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:40.896441 6993 tablet_service.cc:1514] Processing DeleteTablet for tablet 902a8727fa65497aafbc703f3101126b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:40 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:40.896776 7255 tablet_service.cc:1514] Processing DeleteTablet for tablet 556a177350b9483c861972c842a8d193 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:40 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:40.896912 7254 tablet_service.cc:1514] Processing DeleteTablet for tablet 902a8727fa65497aafbc703f3101126b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:40 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:40.897037 7256 tablet_service.cc:1514] Processing DeleteTablet for tablet 249513672b7043d6adcf95ac5ac10a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:40 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:40.898272 10923 ts_tablet_manager.cc:1888] T 556a177350b9483c861972c842a8d193 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:40.898347 10923 log.cc:1196] T 556a177350b9483c861972c842a8d193 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/556a177350b9483c861972c842a8d193
I20240503 01:59:40.898793 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet 902a8727fa65497aafbc703f3101126b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:40 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:40.899000 6863 tablet_service.cc:1514] Processing DeleteTablet for tablet 249513672b7043d6adcf95ac5ac10a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:40 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:40.899133 6992 tablet_service.cc:1514] Processing DeleteTablet for tablet 249513672b7043d6adcf95ac5ac10a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:40 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:40.899300 11055 tablet_replica.cc:330] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:40.899413 11055 raft_consensus.cc:2229] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:40.899626 11055 raft_consensus.cc:2258] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.900054 11055 ts_tablet_manager.cc:1875] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:40.901849 10923 ts_tablet_manager.cc:1909] T 556a177350b9483c861972c842a8d193 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting consensus metadata
I20240503 01:59:40.902617 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet 556a177350b9483c861972c842a8d193 (table rebalance_test_table_18 [id=3a80d416130e4385870aea6e384ed486]) successfully deleted
I20240503 01:59:40.903925 11055 ts_tablet_manager.cc:1888] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:40.904001 11055 log.cc:1196] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/556a177350b9483c861972c842a8d193
I20240503 01:59:40.904258 11055 ts_tablet_manager.cc:1909] T 556a177350b9483c861972c842a8d193 P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:40.904822 11055 tablet_replica.cc:330] T 902a8727fa65497aafbc703f3101126b P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:40.904958 11055 raft_consensus.cc:2229] T 902a8727fa65497aafbc703f3101126b P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.904924 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet 556a177350b9483c861972c842a8d193 (table rebalance_test_table_18 [id=3a80d416130e4385870aea6e384ed486]) successfully deleted
I20240503 01:59:40.905336 11055 raft_consensus.cc:2258] T 902a8727fa65497aafbc703f3101126b P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.906085 11055 ts_tablet_manager.cc:1875] T 902a8727fa65497aafbc703f3101126b P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:40.908428 11055 ts_tablet_manager.cc:1888] T 902a8727fa65497aafbc703f3101126b P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:40.909018 11055 log.cc:1196] T 902a8727fa65497aafbc703f3101126b P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/902a8727fa65497aafbc703f3101126b
I20240503 01:59:40.909363 11055 ts_tablet_manager.cc:1909] T 902a8727fa65497aafbc703f3101126b P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:40.909399 11412 tablet_replica.cc:330] T 556a177350b9483c861972c842a8d193 P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:40.909626 11412 raft_consensus.cc:2229] T 556a177350b9483c861972c842a8d193 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.909893 11412 raft_consensus.cc:2258] T 556a177350b9483c861972c842a8d193 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.910063 11055 tablet_replica.cc:330] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:40.910236 11055 raft_consensus.cc:2229] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:40.910439 11055 raft_consensus.cc:2258] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.910481 11412 ts_tablet_manager.cc:1875] T 556a177350b9483c861972c842a8d193 P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:40.912003 11055 ts_tablet_manager.cc:1875] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:40.912788 11412 ts_tablet_manager.cc:1888] T 556a177350b9483c861972c842a8d193 P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:40.912916 11412 log.cc:1196] T 556a177350b9483c861972c842a8d193 P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/556a177350b9483c861972c842a8d193
I20240503 01:59:40.913146 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet 902a8727fa65497aafbc703f3101126b (table rebalance_test_table_18 [id=3a80d416130e4385870aea6e384ed486]) successfully deleted
I20240503 01:59:40.913223 11412 ts_tablet_manager.cc:1909] T 556a177350b9483c861972c842a8d193 P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:40.913913 11412 tablet_replica.cc:330] T 902a8727fa65497aafbc703f3101126b P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:40.913992 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet 556a177350b9483c861972c842a8d193 (table rebalance_test_table_18 [id=3a80d416130e4385870aea6e384ed486]) successfully deleted
I20240503 01:59:40.914140 11412 raft_consensus.cc:2229] T 902a8727fa65497aafbc703f3101126b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.914157 7124 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:40.914316 11412 raft_consensus.cc:2258] T 902a8727fa65497aafbc703f3101126b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.914651 9736 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:40.914956 11412 ts_tablet_manager.cc:1875] T 902a8727fa65497aafbc703f3101126b P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:40.917312 11412 ts_tablet_manager.cc:1888] T 902a8727fa65497aafbc703f3101126b P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:40.917469 11412 log.cc:1196] T 902a8727fa65497aafbc703f3101126b P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/902a8727fa65497aafbc703f3101126b
I20240503 01:59:40.917793 11412 ts_tablet_manager.cc:1909] T 902a8727fa65497aafbc703f3101126b P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:40.918385 11412 tablet_replica.cc:330] T 249513672b7043d6adcf95ac5ac10a73 P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:40.918491 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet 902a8727fa65497aafbc703f3101126b (table rebalance_test_table_18 [id=3a80d416130e4385870aea6e384ed486]) successfully deleted
I20240503 01:59:40.918648 11412 raft_consensus.cc:2229] T 249513672b7043d6adcf95ac5ac10a73 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.918843 11412 raft_consensus.cc:2258] T 249513672b7043d6adcf95ac5ac10a73 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.919600 11412 ts_tablet_manager.cc:1875] T 249513672b7043d6adcf95ac5ac10a73 P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:40.921608 11412 ts_tablet_manager.cc:1888] T 249513672b7043d6adcf95ac5ac10a73 P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:40.921725 11412 log.cc:1196] T 249513672b7043d6adcf95ac5ac10a73 P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/249513672b7043d6adcf95ac5ac10a73
I20240503 01:59:40.923044 11412 ts_tablet_manager.cc:1909] T 249513672b7043d6adcf95ac5ac10a73 P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:40.927119 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet 249513672b7043d6adcf95ac5ac10a73 (table rebalance_test_table_18 [id=3a80d416130e4385870aea6e384ed486]) successfully deleted
I20240503 01:59:40.928956 6737 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:40.930717 9473 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:40.934743 11094 tablet.cc:1722] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:40.937606 11093 tablet.cc:1722] T 83366a94af3f47caaafd57be6293c138 P 63331f455be049c1bdd6c23c59ac1bef: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:40.942936 11327 tablet.cc:1722] T 83366a94af3f47caaafd57be6293c138 P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:40.943094 6738 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=1848b5ab1d7143aa833d7913acd549fe] alter complete (version 3)
I20240503 01:59:40.949973 11055 ts_tablet_manager.cc:1888] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:40.950075 11055 log.cc:1196] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/249513672b7043d6adcf95ac5ac10a73
I20240503 01:59:40.950418 11055 ts_tablet_manager.cc:1909] T 249513672b7043d6adcf95ac5ac10a73 P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:40.950716 11413 tablet_replica.cc:330] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:40.950935 11413 raft_consensus.cc:2229] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:40.951244 11413 raft_consensus.cc:2258] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.951738 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet 249513672b7043d6adcf95ac5ac10a73 (table rebalance_test_table_18 [id=3a80d416130e4385870aea6e384ed486]) successfully deleted
I20240503 01:59:40.951874 11413 ts_tablet_manager.cc:1875] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:40.954564 11413 ts_tablet_manager.cc:1888] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:40.954674 11413 log.cc:1196] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/902a8727fa65497aafbc703f3101126b
I20240503 01:59:40.962790 11413 ts_tablet_manager.cc:1909] T 902a8727fa65497aafbc703f3101126b P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:40.963794 11413 tablet_replica.cc:330] T 249513672b7043d6adcf95ac5ac10a73 P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:40.963868 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 902a8727fa65497aafbc703f3101126b (table rebalance_test_table_18 [id=3a80d416130e4385870aea6e384ed486]) successfully deleted
I20240503 01:59:40.963958 11413 raft_consensus.cc:2229] T 249513672b7043d6adcf95ac5ac10a73 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:40.964123 11413 raft_consensus.cc:2258] T 249513672b7043d6adcf95ac5ac10a73 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:40.964737 11413 ts_tablet_manager.cc:1875] T 249513672b7043d6adcf95ac5ac10a73 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:40.967432 11413 ts_tablet_manager.cc:1888] T 249513672b7043d6adcf95ac5ac10a73 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 01:59:40.967537 11413 log.cc:1196] T 249513672b7043d6adcf95ac5ac10a73 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/249513672b7043d6adcf95ac5ac10a73
I20240503 01:59:40.967808 11413 ts_tablet_manager.cc:1909] T 249513672b7043d6adcf95ac5ac10a73 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:40.968622 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 249513672b7043d6adcf95ac5ac10a73 (table rebalance_test_table_18 [id=3a80d416130e4385870aea6e384ed486]) successfully deleted
I20240503 01:59:40.991799 7145 consensus_queue.cc:237] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:40.993360 6882 raft_consensus.cc:1261] T 8295ea038a88485095544cc608b000c2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.993410 9626 raft_consensus.cc:1261] T 8295ea038a88485095544cc608b000c2 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:40.993803 10953 consensus_queue.cc:1035] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.994005 9092 consensus_queue.cc:1035] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:40.995836 10953 raft_consensus.cc:2932] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:40.996600 6882 raft_consensus.cc:2932] T 8295ea038a88485095544cc608b000c2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:40.998667 9626 raft_consensus.cc:2932] T 8295ea038a88485095544cc608b000c2 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:40.999158 6738 catalog_manager.cc:5484] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index 4 to 5, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:41.002161 6992 tablet_service.cc:1514] Processing DeleteTablet for tablet 8295ea038a88485095544cc608b000c2 with delete_type TABLET_DATA_TOMBSTONED (TS 63331f455be049c1bdd6c23c59ac1bef not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:41.002415 11055 tablet_replica.cc:330] T 8295ea038a88485095544cc608b000c2 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:41.002604 11055 raft_consensus.cc:2229] T 8295ea038a88485095544cc608b000c2 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:41.002782 11055 raft_consensus.cc:2258] T 8295ea038a88485095544cc608b000c2 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:41.003326 11055 ts_tablet_manager.cc:1875] T 8295ea038a88485095544cc608b000c2 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:41.005700 11055 ts_tablet_manager.cc:1888] T 8295ea038a88485095544cc608b000c2 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 01:59:41.005770 11055 log.cc:1196] T 8295ea038a88485095544cc608b000c2 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/8295ea038a88485095544cc608b000c2
I20240503 01:59:41.006479 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet 8295ea038a88485095544cc608b000c2 (table rebalance_test_table_11 [id=dce2b86d058140cfb0a0d5088d32ec71]) successfully deleted
I20240503 01:59:41.045256 6740 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:41.045516 6740 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:41.048215 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 83366a94af3f47caaafd57be6293c138
I20240503 01:59:41.049458 9604 tablet_service.cc:1514] Processing DeleteTablet for tablet 83366a94af3f47caaafd57be6293c138 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:41 UTC) from {username='slave'} at 127.0.0.1:58562
I20240503 01:59:41.049634 6992 tablet_service.cc:1514] Processing DeleteTablet for tablet 83366a94af3f47caaafd57be6293c138 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:41 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:41.049888 11055 tablet_replica.cc:330] T 83366a94af3f47caaafd57be6293c138 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:41.050024 11055 raft_consensus.cc:2229] T 83366a94af3f47caaafd57be6293c138 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:41.050139 11055 raft_consensus.cc:2258] T 83366a94af3f47caaafd57be6293c138 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:41.050155 9736 tablet_service.cc:1514] Processing DeleteTablet for tablet 83366a94af3f47caaafd57be6293c138 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:41 UTC) from {username='slave'} at 127.0.0.1:48152
I20240503 01:59:41.051304 11471 tablet_replica.cc:330] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191: stopping tablet replica
I20240503 01:59:41.051518 11471 raft_consensus.cc:2229] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:41.051797 11471 raft_consensus.cc:2258] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:41.051847 11055 ts_tablet_manager.cc:1875] T 83366a94af3f47caaafd57be6293c138 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:41.052664 11471 ts_tablet_manager.cc:1875] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:41.054323 11055 ts_tablet_manager.cc:1888] T 83366a94af3f47caaafd57be6293c138 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:41.054415 11055 log.cc:1196] T 83366a94af3f47caaafd57be6293c138 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/83366a94af3f47caaafd57be6293c138
I20240503 01:59:41.054764 11055 ts_tablet_manager.cc:1909] T 83366a94af3f47caaafd57be6293c138 P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:41.055069 11470 tablet_replica.cc:330] T 83366a94af3f47caaafd57be6293c138 P b7062fbbc2b54fff9e99ec4ea5562c76: stopping tablet replica
I20240503 01:59:41.055224 11470 raft_consensus.cc:2229] T 83366a94af3f47caaafd57be6293c138 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:41.055380 11470 raft_consensus.cc:2258] T 83366a94af3f47caaafd57be6293c138 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:41.055923 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet 83366a94af3f47caaafd57be6293c138 (table rebalancer_dynamic_table_new_name [id=1848b5ab1d7143aa833d7913acd549fe]) successfully deleted
I20240503 01:59:41.056377 11470 ts_tablet_manager.cc:1875] T 83366a94af3f47caaafd57be6293c138 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:41.058367 11471 ts_tablet_manager.cc:1888] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:41.058477 11471 log.cc:1196] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/83366a94af3f47caaafd57be6293c138
I20240503 01:59:41.058804 11471 ts_tablet_manager.cc:1909] T 83366a94af3f47caaafd57be6293c138 P db1a0000aaa346f79c0aa4b6c5105191: Deleting consensus metadata
I20240503 01:59:41.058948 11470 ts_tablet_manager.cc:1888] T 83366a94af3f47caaafd57be6293c138 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:41.059033 11470 log.cc:1196] T 83366a94af3f47caaafd57be6293c138 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/83366a94af3f47caaafd57be6293c138
I20240503 01:59:41.059312 11470 ts_tablet_manager.cc:1909] T 83366a94af3f47caaafd57be6293c138 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting consensus metadata
I20240503 01:59:41.059985 6726 catalog_manager.cc:4830] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): tablet 83366a94af3f47caaafd57be6293c138 (table rebalancer_dynamic_table_new_name [id=1848b5ab1d7143aa833d7913acd549fe]) successfully deleted
I20240503 01:59:41.060670 6727 catalog_manager.cc:4830] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): tablet 83366a94af3f47caaafd57be6293c138 (table rebalancer_dynamic_table_new_name [id=1848b5ab1d7143aa833d7913acd549fe]) successfully deleted
I20240503 01:59:41.150578 6740 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:41.157656 9343 tablet_service.cc:1467] Processing CreateTablet for tablet 5c95f8949fbc4650b4d9b4030665d8e7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4aa279e7c78341658a9fa6034d4f4f39]), partition=
I20240503 01:59:41.157733 9736 tablet_service.cc:1467] Processing CreateTablet for tablet 5c95f8949fbc4650b4d9b4030665d8e7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4aa279e7c78341658a9fa6034d4f4f39]), partition=
I20240503 01:59:41.157986 9343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c95f8949fbc4650b4d9b4030665d8e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.158018 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c95f8949fbc4650b4d9b4030665d8e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.161594 11329 tablet_bootstrap.cc:492] T 5c95f8949fbc4650b4d9b4030665d8e7 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:41.162652 11329 tablet_bootstrap.cc:654] T 5c95f8949fbc4650b4d9b4030665d8e7 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:41.163486 9473 tablet_service.cc:1467] Processing CreateTablet for tablet 5c95f8949fbc4650b4d9b4030665d8e7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4aa279e7c78341658a9fa6034d4f4f39]), partition=
I20240503 01:59:41.163754 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c95f8949fbc4650b4d9b4030665d8e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.165552 11274 tablet_bootstrap.cc:492] T 5c95f8949fbc4650b4d9b4030665d8e7 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:41.166635 11274 tablet_bootstrap.cc:654] T 5c95f8949fbc4650b4d9b4030665d8e7 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:41.166683 11333 tablet_bootstrap.cc:492] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:41.167649 11333 tablet_bootstrap.cc:654] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:41.168253 11274 tablet_bootstrap.cc:492] T 5c95f8949fbc4650b4d9b4030665d8e7 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:41.168334 11274 ts_tablet_manager.cc:1367] T 5c95f8949fbc4650b4d9b4030665d8e7 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 01:59:41.168792 11274 raft_consensus.cc:348] T 5c95f8949fbc4650b4d9b4030665d8e7 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:41.168910 11274 raft_consensus.cc:374] T 5c95f8949fbc4650b4d9b4030665d8e7 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:41.168944 11274 raft_consensus.cc:729] T 5c95f8949fbc4650b4d9b4030665d8e7 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:41.169054 11274 consensus_queue.cc:260] T 5c95f8949fbc4650b4d9b4030665d8e7 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:41.169343 11274 ts_tablet_manager.cc:1398] T 5c95f8949fbc4650b4d9b4030665d8e7 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:41.169544 7256 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:41.171298 9604 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:41.173388 10953 consensus_queue.cc:1035] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:41.178560 6992 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:41.179379 7124 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:41.179890 9092 consensus_queue.cc:1035] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:41.185778 11333 tablet_bootstrap.cc:492] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:41.185904 11333 ts_tablet_manager.cc:1367] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 01:59:41.186421 11333 raft_consensus.cc:348] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:41.186570 11333 raft_consensus.cc:374] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:41.186625 11333 raft_consensus.cc:729] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:41.186756 11333 consensus_queue.cc:260] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:41.187036 11333 ts_tablet_manager.cc:1398] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:41.191234 11329 tablet_bootstrap.cc:492] T 5c95f8949fbc4650b4d9b4030665d8e7 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:41.191341 11329 ts_tablet_manager.cc:1367] T 5c95f8949fbc4650b4d9b4030665d8e7 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.002s
I20240503 01:59:41.192087 11329 raft_consensus.cc:348] T 5c95f8949fbc4650b4d9b4030665d8e7 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:41.192201 11329 raft_consensus.cc:374] T 5c95f8949fbc4650b4d9b4030665d8e7 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:41.192229 11329 raft_consensus.cc:729] T 5c95f8949fbc4650b4d9b4030665d8e7 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:41.192348 11329 consensus_queue.cc:260] T 5c95f8949fbc4650b4d9b4030665d8e7 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:41.192600 11329 ts_tablet_manager.cc:1398] T 5c95f8949fbc4650b4d9b4030665d8e7 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:41.195253 6863 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:41.196661 9473 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:41.197713 9343 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:41.198169 9736 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:41.210816 10869 raft_consensus.cc:482] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:41.210979 10869 raft_consensus.cc:504] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:41.211413 10869 leader_election.cc:288] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:41.211964 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c95f8949fbc4650b4d9b4030665d8e7" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:41.212106 9755 raft_consensus.cc:2454] T 5c95f8949fbc4650b4d9b4030665d8e7 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:41.212425 9428 leader_election.cc:302] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:41.212782 10869 raft_consensus.cc:2781] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:41.212833 10869 raft_consensus.cc:482] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:41.212867 10869 raft_consensus.cc:3037] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:41.213972 10869 raft_consensus.cc:504] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:41.214330 10869 leader_election.cc:288] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Requested vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:41.215348 9362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c95f8949fbc4650b4d9b4030665d8e7" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:41.215480 9362 raft_consensus.cc:3037] T 5c95f8949fbc4650b4d9b4030665d8e7 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:41.216527 9362 raft_consensus.cc:2454] T 5c95f8949fbc4650b4d9b4030665d8e7 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:41.217065 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c95f8949fbc4650b4d9b4030665d8e7" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:41.217191 9755 raft_consensus.cc:3037] T 5c95f8949fbc4650b4d9b4030665d8e7 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:41.218217 9755 raft_consensus.cc:2454] T 5c95f8949fbc4650b4d9b4030665d8e7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:41.219067 9428 leader_election.cc:302] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:41.219528 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c95f8949fbc4650b4d9b4030665d8e7" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:41.219684 9363 raft_consensus.cc:2362] T 5c95f8949fbc4650b4d9b4030665d8e7 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1. Re-sending same reply.
I20240503 01:59:41.220145 10869 raft_consensus.cc:2781] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:41.220232 10869 raft_consensus.cc:686] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Becoming Leader. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Running, Role: LEADER
I20240503 01:59:41.220350 10869 consensus_queue.cc:237] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:41.222540 6737 catalog_manager.cc:5484] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: term changed from 0 to 1, leader changed from <none> to bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:41.273859 7145 consensus_queue.cc:237] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:41.275229 9494 raft_consensus.cc:1261] T db656aaa7b104b3182350de320f67839 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:41.275226 7012 raft_consensus.cc:1261] T db656aaa7b104b3182350de320f67839 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:41.275640 10953 consensus_queue.cc:1035] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:41.275828 9092 consensus_queue.cc:1035] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:41.278234 10953 raft_consensus.cc:2932] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:41.278573 9494 raft_consensus.cc:2932] T db656aaa7b104b3182350de320f67839 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:41.278923 7012 raft_consensus.cc:2932] T db656aaa7b104b3182350de320f67839 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:41.281774 6737 catalog_manager.cc:5484] T db656aaa7b104b3182350de320f67839 P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:41.284225 6863 tablet_service.cc:1514] Processing DeleteTablet for tablet db656aaa7b104b3182350de320f67839 with delete_type TABLET_DATA_TOMBSTONED (TS 5dcc33cc621a4cbbae155f3230fb7c46 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:41.284530 11413 tablet_replica.cc:330] T db656aaa7b104b3182350de320f67839 P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:41.284698 11413 raft_consensus.cc:2229] T db656aaa7b104b3182350de320f67839 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:41.284843 11413 raft_consensus.cc:2258] T db656aaa7b104b3182350de320f67839 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:41.285290 11413 ts_tablet_manager.cc:1875] T db656aaa7b104b3182350de320f67839 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:41.287433 11413 ts_tablet_manager.cc:1888] T db656aaa7b104b3182350de320f67839 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 01:59:41.287490 11413 log.cc:1196] T db656aaa7b104b3182350de320f67839 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/db656aaa7b104b3182350de320f67839
I20240503 01:59:41.287948 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet db656aaa7b104b3182350de320f67839 (table rebalance_test_table_15 [id=76d0e687959345af9b8f30fe2c35d640]) successfully deleted
I20240503 01:59:41.313819 10079 consensus_queue.cc:1035] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:41.317742 10855 consensus_queue.cc:1035] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:41.331717 6735 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:41.335990 9755 raft_consensus.cc:1261] T 5c95f8949fbc4650b4d9b4030665d8e7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:41.336225 9363 raft_consensus.cc:1261] T 5c95f8949fbc4650b4d9b4030665d8e7 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:41.336477 10359 consensus_queue.cc:1035] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:41.336808 11573 consensus_queue.cc:1035] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:41.339581 6737 catalog_manager.cc:5576] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053) does not have the latest schema for tablet 5c95f8949fbc4650b4d9b4030665d8e7 (table rebalancer_dynamic_table [id=4aa279e7c78341658a9fa6034d4f4f39]). Expected version 1 got 0
I20240503 01:59:41.339831 10869 consensus_queue.cc:1035] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:41.341025 11577 tablet.cc:1722] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:41.341431 11577 tablet.cc:1620] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:41.341787 6737 catalog_manager.cc:5576] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387) does not have the latest schema for tablet 5c95f8949fbc4650b4d9b4030665d8e7 (table rebalancer_dynamic_table [id=4aa279e7c78341658a9fa6034d4f4f39]). Expected version 1 got 0
I20240503 01:59:41.346225 6738 catalog_manager.cc:5576] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387) does not have the latest schema for tablet 5c95f8949fbc4650b4d9b4030665d8e7 (table rebalancer_dynamic_table [id=4aa279e7c78341658a9fa6034d4f4f39]). Expected version 1 got 0
I20240503 01:59:41.346802 11094 tablet.cc:1722] T 5c95f8949fbc4650b4d9b4030665d8e7 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:41.347123 11094 tablet.cc:1620] T 5c95f8949fbc4650b4d9b4030665d8e7 P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:41.348311 6735 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=4aa279e7c78341658a9fa6034d4f4f39] alter complete (version 1)
I20240503 01:59:41.345021 11578 tablet.cc:1722] T 5c95f8949fbc4650b4d9b4030665d8e7 P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:41.354045 11578 tablet.cc:1620] T 5c95f8949fbc4650b4d9b4030665d8e7 P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 01:59:41.361714 10869 consensus_queue.cc:1035] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:41.387054 6734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:41.395354 9343 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:41.396705 9604 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:41.397176 9343 tablet_service.cc:1467] Processing CreateTablet for tablet 9b9405669fb640c4a3a8fe2f31af291e (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=d46b94d9de0c43ad876391aee881e04d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:41.397524 9343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b9405669fb640c4a3a8fe2f31af291e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.399417 6992 tablet_service.cc:1467] Processing CreateTablet for tablet 487eafe47df84b468618f473eca37874 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=d46b94d9de0c43ad876391aee881e04d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:41.399689 6992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 487eafe47df84b468618f473eca37874. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.400012 11329 tablet_bootstrap.cc:492] T 9b9405669fb640c4a3a8fe2f31af291e P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:41.400696 9736 tablet_service.cc:1467] Processing CreateTablet for tablet 487eafe47df84b468618f473eca37874 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=d46b94d9de0c43ad876391aee881e04d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:41.400949 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 487eafe47df84b468618f473eca37874. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.401011 11329 tablet_bootstrap.cc:654] T 9b9405669fb640c4a3a8fe2f31af291e P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:41.401847 6863 tablet_service.cc:1467] Processing CreateTablet for tablet 9b9405669fb640c4a3a8fe2f31af291e (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=d46b94d9de0c43ad876391aee881e04d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:41.402107 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b9405669fb640c4a3a8fe2f31af291e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.402673 11609 tablet_bootstrap.cc:492] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:41.402792 9737 tablet_service.cc:1467] Processing CreateTablet for tablet fad1113f16ba45009ae409f57eac4550 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=d46b94d9de0c43ad876391aee881e04d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:41.403012 9737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fad1113f16ba45009ae409f57eac4550. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.403636 11609 tablet_bootstrap.cc:654] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:41.405167 11274 tablet_bootstrap.cc:492] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:41.406181 11274 tablet_bootstrap.cc:654] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:41.406755 6863 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:41.406276 11609 tablet_bootstrap.cc:492] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:41.407979 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 487eafe47df84b468618f473eca37874 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=d46b94d9de0c43ad876391aee881e04d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:41.408252 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 487eafe47df84b468618f473eca37874. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.409092 11609 ts_tablet_manager.cc:1367] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:41.409191 9604 tablet_service.cc:1467] Processing CreateTablet for tablet 9b9405669fb640c4a3a8fe2f31af291e (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=d46b94d9de0c43ad876391aee881e04d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:41.409498 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b9405669fb640c4a3a8fe2f31af291e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.409616 7256 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:41.410672 9341 tablet_service.cc:1467] Processing CreateTablet for tablet fad1113f16ba45009ae409f57eac4550 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=d46b94d9de0c43ad876391aee881e04d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:41.410758 11609 raft_consensus.cc:348] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:41.410881 11609 raft_consensus.cc:374] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:41.410913 11609 raft_consensus.cc:729] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:41.410949 9341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fad1113f16ba45009ae409f57eac4550. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.411103 11609 consensus_queue.cc:260] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:41.412045 9605 tablet_service.cc:1467] Processing CreateTablet for tablet fad1113f16ba45009ae409f57eac4550 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=d46b94d9de0c43ad876391aee881e04d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:41.412092 11609 ts_tablet_manager.cc:1398] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:41.412285 9605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fad1113f16ba45009ae409f57eac4550. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.417603 11613 tablet_bootstrap.cc:492] T 9b9405669fb640c4a3a8fe2f31af291e P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:41.418759 11613 tablet_bootstrap.cc:654] T 9b9405669fb640c4a3a8fe2f31af291e P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:41.419723 9473 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:41.419806 11274 tablet_bootstrap.cc:492] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:41.419931 11274 ts_tablet_manager.cc:1367] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.002s
I20240503 01:59:41.420490 11274 raft_consensus.cc:348] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:41.420616 11274 raft_consensus.cc:374] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:41.420648 11274 raft_consensus.cc:729] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:41.420774 11274 consensus_queue.cc:260] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:41.420805 7124 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:41.421061 11274 ts_tablet_manager.cc:1398] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:41.429570 11274 tablet_bootstrap.cc:492] T 487eafe47df84b468618f473eca37874 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:41.433705 11274 tablet_bootstrap.cc:654] T 487eafe47df84b468618f473eca37874 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:41.449968 6741 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:41.454186 11274 tablet_bootstrap.cc:492] T 487eafe47df84b468618f473eca37874 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:41.456259 11274 ts_tablet_manager.cc:1367] T 487eafe47df84b468618f473eca37874 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.003s
I20240503 01:59:41.456287 11577 tablet.cc:1722] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:41.456666 11577 tablet.cc:1620] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:41.456832 11274 raft_consensus.cc:348] T 487eafe47df84b468618f473eca37874 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:41.456941 11274 raft_consensus.cc:374] T 487eafe47df84b468618f473eca37874 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:41.456969 11274 raft_consensus.cc:729] T 487eafe47df84b468618f473eca37874 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:41.459520 11094 tablet.cc:1722] T 5c95f8949fbc4650b4d9b4030665d8e7 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:41.459884 11094 tablet.cc:1620] T 5c95f8949fbc4650b4d9b4030665d8e7 P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:41.460005 11578 tablet.cc:1722] T 5c95f8949fbc4650b4d9b4030665d8e7 P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:41.460464 11578 tablet.cc:1620] T 5c95f8949fbc4650b4d9b4030665d8e7 P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 01:59:41.462297 11274 consensus_queue.cc:260] T 487eafe47df84b468618f473eca37874 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:41.462747 11274 ts_tablet_manager.cc:1398] T 487eafe47df84b468618f473eca37874 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240503 01:59:41.463864 9736 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:41.464143 6741 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=4aa279e7c78341658a9fa6034d4f4f39] alter complete (version 2)
I20240503 01:59:41.466646 11611 tablet_bootstrap.cc:492] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:41.467638 11613 tablet_bootstrap.cc:492] T 9b9405669fb640c4a3a8fe2f31af291e P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:41.467729 11613 ts_tablet_manager.cc:1367] T 9b9405669fb640c4a3a8fe2f31af291e P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.000s
I20240503 01:59:41.467821 11329 tablet_bootstrap.cc:492] T 9b9405669fb640c4a3a8fe2f31af291e P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:41.467901 11329 ts_tablet_manager.cc:1367] T 9b9405669fb640c4a3a8fe2f31af291e P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.068s user 0.000s sys 0.002s
I20240503 01:59:41.468358 11329 raft_consensus.cc:348] T 9b9405669fb640c4a3a8fe2f31af291e P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:41.468468 11329 raft_consensus.cc:374] T 9b9405669fb640c4a3a8fe2f31af291e P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:41.468497 11329 raft_consensus.cc:729] T 9b9405669fb640c4a3a8fe2f31af291e P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:41.468448 11613 raft_consensus.cc:348] T 9b9405669fb640c4a3a8fe2f31af291e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:41.468640 11613 raft_consensus.cc:374] T 9b9405669fb640c4a3a8fe2f31af291e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:41.468619 11329 consensus_queue.cc:260] T 9b9405669fb640c4a3a8fe2f31af291e P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:41.468674 11613 raft_consensus.cc:729] T 9b9405669fb640c4a3a8fe2f31af291e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:41.468796 11613 consensus_queue.cc:260] T 9b9405669fb640c4a3a8fe2f31af291e P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:41.468911 11329 ts_tablet_manager.cc:1398] T 9b9405669fb640c4a3a8fe2f31af291e P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:41.469070 11613 ts_tablet_manager.cc:1398] T 9b9405669fb640c4a3a8fe2f31af291e P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:41.469092 11329 tablet_bootstrap.cc:492] T 487eafe47df84b468618f473eca37874 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:41.469215 11613 tablet_bootstrap.cc:492] T fad1113f16ba45009ae409f57eac4550 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:41.470045 11329 tablet_bootstrap.cc:654] T 487eafe47df84b468618f473eca37874 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:41.470247 11613 tablet_bootstrap.cc:654] T fad1113f16ba45009ae409f57eac4550 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:41.471014 6992 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:41.472926 11611 tablet_bootstrap.cc:654] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:41.483525 11329 tablet_bootstrap.cc:492] T 487eafe47df84b468618f473eca37874 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:41.483661 11329 ts_tablet_manager.cc:1367] T 487eafe47df84b468618f473eca37874 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.001s
I20240503 01:59:41.483697 11613 tablet_bootstrap.cc:492] T fad1113f16ba45009ae409f57eac4550 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:41.483780 11613 ts_tablet_manager.cc:1367] T fad1113f16ba45009ae409f57eac4550 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 01:59:41.484220 11329 raft_consensus.cc:348] T 487eafe47df84b468618f473eca37874 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:41.484202 11613 raft_consensus.cc:348] T fad1113f16ba45009ae409f57eac4550 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:41.484320 11613 raft_consensus.cc:374] T fad1113f16ba45009ae409f57eac4550 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:41.484320 11329 raft_consensus.cc:374] T 487eafe47df84b468618f473eca37874 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:41.484352 11613 raft_consensus.cc:729] T fad1113f16ba45009ae409f57eac4550 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:41.484352 11329 raft_consensus.cc:729] T 487eafe47df84b468618f473eca37874 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:41.484489 11329 consensus_queue.cc:260] T 487eafe47df84b468618f473eca37874 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:41.484633 11613 consensus_queue.cc:260] T fad1113f16ba45009ae409f57eac4550 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:41.484815 11329 ts_tablet_manager.cc:1398] T 487eafe47df84b468618f473eca37874 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:41.484906 11613 ts_tablet_manager.cc:1398] T fad1113f16ba45009ae409f57eac4550 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:41.484995 11329 tablet_bootstrap.cc:492] T fad1113f16ba45009ae409f57eac4550 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:41.486016 11329 tablet_bootstrap.cc:654] T fad1113f16ba45009ae409f57eac4550 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:41.488241 11611 tablet_bootstrap.cc:492] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:41.488399 11611 ts_tablet_manager.cc:1367] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240503 01:59:41.488890 11611 raft_consensus.cc:348] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:41.489102 11611 raft_consensus.cc:374] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:41.489185 11611 raft_consensus.cc:729] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:41.489369 11611 consensus_queue.cc:260] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:41.489691 11611 ts_tablet_manager.cc:1398] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:41.491153 11329 tablet_bootstrap.cc:492] T fad1113f16ba45009ae409f57eac4550 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:41.491254 11329 ts_tablet_manager.cc:1367] T fad1113f16ba45009ae409f57eac4550 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20240503 01:59:41.491350 10855 raft_consensus.cc:482] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:41.491446 10855 raft_consensus.cc:504] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:41.491732 11329 raft_consensus.cc:348] T fad1113f16ba45009ae409f57eac4550 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:41.491829 11329 raft_consensus.cc:374] T fad1113f16ba45009ae409f57eac4550 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:41.491852 10855 leader_election.cc:288] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:41.491865 11329 raft_consensus.cc:729] T fad1113f16ba45009ae409f57eac4550 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:41.491981 11329 consensus_queue.cc:260] T fad1113f16ba45009ae409f57eac4550 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:41.492281 11329 ts_tablet_manager.cc:1398] T fad1113f16ba45009ae409f57eac4550 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:41.492252 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "487eafe47df84b468618f473eca37874" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:41.492386 9757 raft_consensus.cc:2454] T 487eafe47df84b468618f473eca37874 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:41.492902 6948 leader_election.cc:302] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:41.493119 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "487eafe47df84b468618f473eca37874" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:41.493219 9363 raft_consensus.cc:2454] T 487eafe47df84b468618f473eca37874 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:41.493299 10079 raft_consensus.cc:2781] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:41.493346 10079 raft_consensus.cc:482] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:41.493392 10079 raft_consensus.cc:3037] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:41.494468 10079 raft_consensus.cc:504] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:41.494827 10079 leader_election.cc:288] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:41.495246 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "487eafe47df84b468618f473eca37874" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:41.495342 9363 raft_consensus.cc:3037] T 487eafe47df84b468618f473eca37874 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:41.496331 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "487eafe47df84b468618f473eca37874" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:41.496371 9363 raft_consensus.cc:2454] T 487eafe47df84b468618f473eca37874 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:41.496443 9757 raft_consensus.cc:3037] T 487eafe47df84b468618f473eca37874 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:41.497005 6949 leader_election.cc:302] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:41.497193 10855 raft_consensus.cc:2781] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:41.497257 10855 raft_consensus.cc:686] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:41.497432 9757 raft_consensus.cc:2454] T 487eafe47df84b468618f473eca37874 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:41.497402 10855 consensus_queue.cc:237] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:41.499893 6734 catalog_manager.cc:5484] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:41.535266 7275 consensus_queue.cc:237] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:41.536569 9757 raft_consensus.cc:1261] T 4600418bf64c4a0d962831c6924c46e1 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:41.536657 6881 raft_consensus.cc:1261] T 4600418bf64c4a0d962831c6924c46e1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:41.536942 11114 consensus_queue.cc:1035] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:41.537159 10585 consensus_queue.cc:1035] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:41.538801 10585 raft_consensus.cc:2932] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:41.539114 9757 raft_consensus.cc:2932] T 4600418bf64c4a0d962831c6924c46e1 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:41.542096 6734 catalog_manager.cc:5484] T 4600418bf64c4a0d962831c6924c46e1 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:41.544582 7124 tablet_service.cc:1514] Processing DeleteTablet for tablet 4600418bf64c4a0d962831c6924c46e1 with delete_type TABLET_DATA_TOMBSTONED (TS c688f3df56ac4b6d9b0c30fa4cc8ca2a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:41.544971 11640 tablet_replica.cc:330] T 4600418bf64c4a0d962831c6924c46e1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:41.544873 6881 raft_consensus.cc:2932] T 4600418bf64c4a0d962831c6924c46e1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:41.545118 11640 raft_consensus.cc:2229] T 4600418bf64c4a0d962831c6924c46e1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:41.545274 11640 raft_consensus.cc:2258] T 4600418bf64c4a0d962831c6924c46e1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:41.545872 11640 ts_tablet_manager.cc:1875] T 4600418bf64c4a0d962831c6924c46e1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:41.548381 11640 ts_tablet_manager.cc:1888] T 4600418bf64c4a0d962831c6924c46e1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 01:59:41.548457 11640 log.cc:1196] T 4600418bf64c4a0d962831c6924c46e1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/4600418bf64c4a0d962831c6924c46e1
I20240503 01:59:41.549660 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet 4600418bf64c4a0d962831c6924c46e1 (table rebalance_test_table_9 [id=f050ea79f4334dcda01dce5d3e804176]) successfully deleted
I20240503 01:59:41.565845 11127 raft_consensus.cc:482] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:41.565972 11127 raft_consensus.cc:504] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:41.566344 11127 leader_election.cc:288] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:41.566757 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad1113f16ba45009ae409f57eac4550" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:41.566840 9626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad1113f16ba45009ae409f57eac4550" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:41.566903 9363 raft_consensus.cc:2454] T fad1113f16ba45009ae409f57eac4550 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 01:59:41.566977 9626 raft_consensus.cc:2454] T fad1113f16ba45009ae409f57eac4550 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 01:59:41.567242 9692 leader_election.cc:302] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:41.567399 11127 raft_consensus.cc:2781] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:41.567458 11127 raft_consensus.cc:482] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:41.567497 11127 raft_consensus.cc:3037] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:41.568358 11127 raft_consensus.cc:504] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:41.568683 11127 leader_election.cc:288] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Requested vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:41.569046 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad1113f16ba45009ae409f57eac4550" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:41.569175 9363 raft_consensus.cc:3037] T fad1113f16ba45009ae409f57eac4550 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:41.569955 9363 raft_consensus.cc:2454] T fad1113f16ba45009ae409f57eac4550 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:41.570222 9692 leader_election.cc:302] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:41.570380 11127 raft_consensus.cc:2781] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:41.570452 11127 raft_consensus.cc:686] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Becoming Leader. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Running, Role: LEADER
I20240503 01:59:41.570631 11127 consensus_queue.cc:237] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:41.570814 9626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad1113f16ba45009ae409f57eac4550" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:41.570935 9626 raft_consensus.cc:3037] T fad1113f16ba45009ae409f57eac4550 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:41.571924 9626 raft_consensus.cc:2454] T fad1113f16ba45009ae409f57eac4550 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:41.573083 6741 catalog_manager.cc:5484] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: term changed from 0 to 1, leader changed from <none> to db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:41.579623 6741 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:41.585893 11577 tablet.cc:1722] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:41.586637 11578 tablet.cc:1722] T 5c95f8949fbc4650b4d9b4030665d8e7 P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:41.586787 11094 tablet.cc:1722] T 5c95f8949fbc4650b4d9b4030665d8e7 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:41.592921 6741 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=4aa279e7c78341658a9fa6034d4f4f39] alter complete (version 3)
I20240503 01:59:41.694033 6741 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:41.694193 6741 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:41.696437 6741 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 5c95f8949fbc4650b4d9b4030665d8e7
I20240503 01:59:41.697007 9736 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c95f8949fbc4650b4d9b4030665d8e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:41 UTC) from {username='slave'} at 127.0.0.1:48152
I20240503 01:59:41.697211 9473 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c95f8949fbc4650b4d9b4030665d8e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:41 UTC) from {username='slave'} at 127.0.0.1:54678
I20240503 01:59:41.697316 9341 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c95f8949fbc4650b4d9b4030665d8e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:41 UTC) from {username='slave'} at 127.0.0.1:36040
I20240503 01:59:41.697548 11682 tablet_replica.cc:330] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a: stopping tablet replica
I20240503 01:59:41.697702 11682 raft_consensus.cc:2229] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:41.697857 11683 tablet_replica.cc:330] T 5c95f8949fbc4650b4d9b4030665d8e7 P 12e8c683db054df3b75c6285b98a5ad8: stopping tablet replica
I20240503 01:59:41.697988 11682 raft_consensus.cc:2258] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:41.698004 11681 tablet_replica.cc:330] T 5c95f8949fbc4650b4d9b4030665d8e7 P db1a0000aaa346f79c0aa4b6c5105191: stopping tablet replica
I20240503 01:59:41.698016 11683 raft_consensus.cc:2229] T 5c95f8949fbc4650b4d9b4030665d8e7 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:41.698163 11683 raft_consensus.cc:2258] T 5c95f8949fbc4650b4d9b4030665d8e7 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:41.698186 11681 raft_consensus.cc:2229] T 5c95f8949fbc4650b4d9b4030665d8e7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:41.698470 11681 raft_consensus.cc:2258] T 5c95f8949fbc4650b4d9b4030665d8e7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:41.698832 11682 ts_tablet_manager.cc:1875] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:41.699095 11683 ts_tablet_manager.cc:1875] T 5c95f8949fbc4650b4d9b4030665d8e7 P 12e8c683db054df3b75c6285b98a5ad8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:41.699319 11681 ts_tablet_manager.cc:1875] T 5c95f8949fbc4650b4d9b4030665d8e7 P db1a0000aaa346f79c0aa4b6c5105191: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:41.701289 11682 ts_tablet_manager.cc:1888] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:41.701289 11683 ts_tablet_manager.cc:1888] T 5c95f8949fbc4650b4d9b4030665d8e7 P 12e8c683db054df3b75c6285b98a5ad8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:41.701371 11683 log.cc:1196] T 5c95f8949fbc4650b4d9b4030665d8e7 P 12e8c683db054df3b75c6285b98a5ad8: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/5c95f8949fbc4650b4d9b4030665d8e7
I20240503 01:59:41.701371 11682 log.cc:1196] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/5c95f8949fbc4650b4d9b4030665d8e7
I20240503 01:59:41.701493 11681 ts_tablet_manager.cc:1888] T 5c95f8949fbc4650b4d9b4030665d8e7 P db1a0000aaa346f79c0aa4b6c5105191: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:41.701579 11681 log.cc:1196] T 5c95f8949fbc4650b4d9b4030665d8e7 P db1a0000aaa346f79c0aa4b6c5105191: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/5c95f8949fbc4650b4d9b4030665d8e7
I20240503 01:59:41.701661 11683 ts_tablet_manager.cc:1909] T 5c95f8949fbc4650b4d9b4030665d8e7 P 12e8c683db054df3b75c6285b98a5ad8: Deleting consensus metadata
I20240503 01:59:41.701663 11682 ts_tablet_manager.cc:1909] T 5c95f8949fbc4650b4d9b4030665d8e7 P bb5364d730cb4489a0bfc6c270b7450a: Deleting consensus metadata
I20240503 01:59:41.701850 11681 ts_tablet_manager.cc:1909] T 5c95f8949fbc4650b4d9b4030665d8e7 P db1a0000aaa346f79c0aa4b6c5105191: Deleting consensus metadata
I20240503 01:59:41.702409 6727 catalog_manager.cc:4830] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): tablet 5c95f8949fbc4650b4d9b4030665d8e7 (table rebalancer_dynamic_table_new_name [id=4aa279e7c78341658a9fa6034d4f4f39]) successfully deleted
I20240503 01:59:41.702593 6725 catalog_manager.cc:4830] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): tablet 5c95f8949fbc4650b4d9b4030665d8e7 (table rebalancer_dynamic_table_new_name [id=4aa279e7c78341658a9fa6034d4f4f39]) successfully deleted
I20240503 01:59:41.702601 6726 catalog_manager.cc:4830] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): tablet 5c95f8949fbc4650b4d9b4030665d8e7 (table rebalancer_dynamic_table_new_name [id=4aa279e7c78341658a9fa6034d4f4f39]) successfully deleted
I20240503 01:59:41.704295 11520 raft_consensus.cc:482] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:41.704434 11520 raft_consensus.cc:504] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:41.705155 11520 leader_election.cc:288] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:41.709930 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b9405669fb640c4a3a8fe2f31af291e" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:41.710152 9363 raft_consensus.cc:2454] T 9b9405669fb640c4a3a8fe2f31af291e P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:41.710608 6818 leader_election.cc:302] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:41.710850 11520 raft_consensus.cc:2781] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:41.710963 11520 raft_consensus.cc:482] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:41.711010 11520 raft_consensus.cc:3037] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:41.711859 9626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b9405669fb640c4a3a8fe2f31af291e" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:41.712038 9626 raft_consensus.cc:2454] T 9b9405669fb640c4a3a8fe2f31af291e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:41.712044 11520 raft_consensus.cc:504] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:41.712405 11520 leader_election.cc:288] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Requested vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:41.712790 9626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b9405669fb640c4a3a8fe2f31af291e" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:41.712846 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b9405669fb640c4a3a8fe2f31af291e" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:41.712895 9626 raft_consensus.cc:3037] T 9b9405669fb640c4a3a8fe2f31af291e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:41.712983 9363 raft_consensus.cc:3037] T 9b9405669fb640c4a3a8fe2f31af291e P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:41.713678 9626 raft_consensus.cc:2454] T 9b9405669fb640c4a3a8fe2f31af291e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:41.713742 9363 raft_consensus.cc:2454] T 9b9405669fb640c4a3a8fe2f31af291e P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:41.714010 6818 leader_election.cc:302] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:41.714191 11520 raft_consensus.cc:2781] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:41.714313 11520 raft_consensus.cc:686] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 01:59:41.714473 11520 consensus_queue.cc:237] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:41.716856 6740 catalog_manager.cc:5484] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:41.736991 9341 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:41.745247 7256 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:41.748109 6992 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:41.748580 9605 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:41.758720 9473 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:41.761953 6863 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:41.769395 9736 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:41.797736 7124 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:41.798708 6740 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:41.805320 9605 tablet_service.cc:1467] Processing CreateTablet for tablet a080239d59ae4fd1be9bb799871130f1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7bc73d615e1b43e39c08cc44a0cbac53]), partition=
I20240503 01:59:41.805320 6863 tablet_service.cc:1467] Processing CreateTablet for tablet a080239d59ae4fd1be9bb799871130f1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7bc73d615e1b43e39c08cc44a0cbac53]), partition=
I20240503 01:59:41.805621 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a080239d59ae4fd1be9bb799871130f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.805631 9605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a080239d59ae4fd1be9bb799871130f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.807904 11611 tablet_bootstrap.cc:492] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:41.807927 11613 tablet_bootstrap.cc:492] T a080239d59ae4fd1be9bb799871130f1 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:41.808840 11613 tablet_bootstrap.cc:654] T a080239d59ae4fd1be9bb799871130f1 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:41.808840 11611 tablet_bootstrap.cc:654] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:41.810262 11611 tablet_bootstrap.cc:492] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:41.810354 11611 ts_tablet_manager.cc:1367] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:41.810488 7256 tablet_service.cc:1467] Processing CreateTablet for tablet a080239d59ae4fd1be9bb799871130f1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7bc73d615e1b43e39c08cc44a0cbac53]), partition=
I20240503 01:59:41.810797 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a080239d59ae4fd1be9bb799871130f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:41.810922 11611 raft_consensus.cc:348] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:41.811053 11611 raft_consensus.cc:374] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:41.811093 11611 raft_consensus.cc:729] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:41.811229 11611 consensus_queue.cc:260] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:41.811796 11613 tablet_bootstrap.cc:492] T a080239d59ae4fd1be9bb799871130f1 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:41.811883 11613 ts_tablet_manager.cc:1367] T a080239d59ae4fd1be9bb799871130f1 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:41.811954 11611 ts_tablet_manager.cc:1398] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:41.812338 11613 raft_consensus.cc:348] T a080239d59ae4fd1be9bb799871130f1 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:41.812454 11613 raft_consensus.cc:374] T a080239d59ae4fd1be9bb799871130f1 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:41.812492 11613 raft_consensus.cc:729] T a080239d59ae4fd1be9bb799871130f1 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:41.812629 11613 consensus_queue.cc:260] T a080239d59ae4fd1be9bb799871130f1 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:41.812889 11613 ts_tablet_manager.cc:1398] T a080239d59ae4fd1be9bb799871130f1 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:41.816220 11700 tablet_bootstrap.cc:492] T a080239d59ae4fd1be9bb799871130f1 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:41.818074 11700 tablet_bootstrap.cc:654] T a080239d59ae4fd1be9bb799871130f1 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:41.818656 11520 raft_consensus.cc:482] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:41.818766 11520 raft_consensus.cc:504] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:41.819159 11520 leader_election.cc:288] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:41.819499 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a080239d59ae4fd1be9bb799871130f1" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:41.820178 11700 tablet_bootstrap.cc:492] T a080239d59ae4fd1be9bb799871130f1 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:41.820219 9626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a080239d59ae4fd1be9bb799871130f1" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:41.820273 11700 ts_tablet_manager.cc:1367] T a080239d59ae4fd1be9bb799871130f1 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 01:59:41.820358 9626 raft_consensus.cc:2454] T a080239d59ae4fd1be9bb799871130f1 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:41.820676 6818 leader_election.cc:302] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:41.820688 11700 raft_consensus.cc:348] T a080239d59ae4fd1be9bb799871130f1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:41.820783 11700 raft_consensus.cc:374] T a080239d59ae4fd1be9bb799871130f1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:41.820814 11700 raft_consensus.cc:729] T a080239d59ae4fd1be9bb799871130f1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
W20240503 01:59:41.820809 6817 leader_election.cc:341] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): Illegal state: must be running to vote when last-logged opid is not known
I20240503 01:59:41.820868 11520 raft_consensus.cc:2781] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:41.820909 11520 raft_consensus.cc:482] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:41.820942 11520 raft_consensus.cc:3037] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:41.820960 11700 consensus_queue.cc:260] T a080239d59ae4fd1be9bb799871130f1 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:41.821241 11700 ts_tablet_manager.cc:1398] T a080239d59ae4fd1be9bb799871130f1 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:41.822088 11520 raft_consensus.cc:504] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:41.822425 11520 leader_election.cc:288] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:41.822814 9626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a080239d59ae4fd1be9bb799871130f1" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:41.822939 9626 raft_consensus.cc:3037] T a080239d59ae4fd1be9bb799871130f1 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:41.823865 9626 raft_consensus.cc:2454] T a080239d59ae4fd1be9bb799871130f1 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:41.824137 6818 leader_election.cc:302] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:41.824235 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a080239d59ae4fd1be9bb799871130f1" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:41.824342 7275 raft_consensus.cc:3037] T a080239d59ae4fd1be9bb799871130f1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:41.824383 11520 raft_consensus.cc:2781] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:41.824455 11520 raft_consensus.cc:686] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 01:59:41.824589 11520 consensus_queue.cc:237] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:41.825264 7275 raft_consensus.cc:2454] T a080239d59ae4fd1be9bb799871130f1 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:41.826457 6741 catalog_manager.cc:5484] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:41.859354 7275 consensus_queue.cc:237] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:41.860724 7143 raft_consensus.cc:1261] T e811662836164c3fb29a1b12ace7eb41 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:41.860723 9494 raft_consensus.cc:1261] T e811662836164c3fb29a1b12ace7eb41 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:41.861186 11114 consensus_queue.cc:1035] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:41.861392 10585 consensus_queue.cc:1035] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:41.863255 10585 raft_consensus.cc:2932] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:41.863694 7143 raft_consensus.cc:2932] T e811662836164c3fb29a1b12ace7eb41 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:41.864040 9494 raft_consensus.cc:2932] T e811662836164c3fb29a1b12ace7eb41 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:41.866768 6741 catalog_manager.cc:5484] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:41.870777 6863 tablet_service.cc:1514] Processing DeleteTablet for tablet e811662836164c3fb29a1b12ace7eb41 with delete_type TABLET_DATA_TOMBSTONED (TS 5dcc33cc621a4cbbae155f3230fb7c46 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:41.871194 11726 tablet_replica.cc:330] T e811662836164c3fb29a1b12ace7eb41 P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:41.871344 11726 raft_consensus.cc:2229] T e811662836164c3fb29a1b12ace7eb41 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:41.871512 11726 raft_consensus.cc:2258] T e811662836164c3fb29a1b12ace7eb41 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:41.872032 11726 ts_tablet_manager.cc:1875] T e811662836164c3fb29a1b12ace7eb41 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:41.874366 11726 ts_tablet_manager.cc:1888] T e811662836164c3fb29a1b12ace7eb41 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 01:59:41.874471 11726 log.cc:1196] T e811662836164c3fb29a1b12ace7eb41 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/e811662836164c3fb29a1b12ace7eb41
I20240503 01:59:41.875170 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet e811662836164c3fb29a1b12ace7eb41 (table rebalance_test_table_17 [id=b061a61957e74150a4af26c16c277330]) successfully deleted
I20240503 01:59:41.933952 6741 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:41.943799 7275 raft_consensus.cc:1261] T a080239d59ae4fd1be9bb799871130f1 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:41.944257 11520 consensus_queue.cc:1035] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:41.944777 9626 raft_consensus.cc:1261] T a080239d59ae4fd1be9bb799871130f1 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:41.945344 10668 consensus_queue.cc:1035] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:41.947233 6734 catalog_manager.cc:5576] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271) does not have the latest schema for tablet a080239d59ae4fd1be9bb799871130f1 (table rebalancer_dynamic_table [id=7bc73d615e1b43e39c08cc44a0cbac53]). Expected version 1 got 0
I20240503 01:59:41.948724 6734 catalog_manager.cc:5576] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087) does not have the latest schema for tablet a080239d59ae4fd1be9bb799871130f1 (table rebalancer_dynamic_table [id=7bc73d615e1b43e39c08cc44a0cbac53]). Expected version 1 got 0
I20240503 01:59:41.952494 11766 tablet.cc:1722] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:41.952162 11767 tablet.cc:1722] T a080239d59ae4fd1be9bb799871130f1 P 1d0e738c34af40fa9950c6f086619b3a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:41.952867 11767 tablet.cc:1620] T a080239d59ae4fd1be9bb799871130f1 P 1d0e738c34af40fa9950c6f086619b3a: MemRowSet was empty: no flush needed.
I20240503 01:59:41.953341 6734 catalog_manager.cc:5576] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271) does not have the latest schema for tablet a080239d59ae4fd1be9bb799871130f1 (table rebalancer_dynamic_table [id=7bc73d615e1b43e39c08cc44a0cbac53]). Expected version 1 got 0
I20240503 01:59:41.954092 11766 tablet.cc:1620] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46: MemRowSet was empty: no flush needed.
I20240503 01:59:41.956162 6734 catalog_manager.cc:5576] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087) does not have the latest schema for tablet a080239d59ae4fd1be9bb799871130f1 (table rebalancer_dynamic_table [id=7bc73d615e1b43e39c08cc44a0cbac53]). Expected version 1 got 0
I20240503 01:59:41.959288 11776 tablet.cc:1722] T a080239d59ae4fd1be9bb799871130f1 P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:41.960009 11776 tablet.cc:1620] T a080239d59ae4fd1be9bb799871130f1 P b7062fbbc2b54fff9e99ec4ea5562c76: MemRowSet was empty: no flush needed.
I20240503 01:59:41.968350 6734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=7bc73d615e1b43e39c08cc44a0cbac53] alter complete (version 1)
I20240503 01:59:41.974103 7256 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:41.974104 9341 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:41.974799 6992 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:41.979372 9605 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:41.984238 6863 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:41.991721 10079 consensus_queue.cc:1035] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:41.993392 7124 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:41.996730 11620 consensus_queue.cc:1035] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:42.004408 9473 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:42.023033 9736 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:42.050525 11127 consensus_queue.cc:1035] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:42.054677 11326 consensus_queue.cc:1035] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:42.075240 6738 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:42.079752 7143 consensus_queue.cc:237] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:42.081646 7012 raft_consensus.cc:1261] T 8ed2793842464443826f83f60fb6dcd1 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:42.082271 11766 tablet.cc:1722] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:42.082690 11766 tablet.cc:1620] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46: MemRowSet was empty: no flush needed.
I20240503 01:59:42.083133 11803 consensus_queue.cc:1035] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:42.083451 9494 raft_consensus.cc:1261] T 8ed2793842464443826f83f60fb6dcd1 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:42.084908 11803 consensus_queue.cc:1035] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:42.085325 11767 tablet.cc:1722] T a080239d59ae4fd1be9bb799871130f1 P 1d0e738c34af40fa9950c6f086619b3a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:42.085670 11767 tablet.cc:1620] T a080239d59ae4fd1be9bb799871130f1 P 1d0e738c34af40fa9950c6f086619b3a: MemRowSet was empty: no flush needed.
I20240503 01:59:42.085899 11776 tablet.cc:1722] T a080239d59ae4fd1be9bb799871130f1 P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:42.086236 11776 tablet.cc:1620] T a080239d59ae4fd1be9bb799871130f1 P b7062fbbc2b54fff9e99ec4ea5562c76: MemRowSet was empty: no flush needed.
I20240503 01:59:42.094240 6738 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=7bc73d615e1b43e39c08cc44a0cbac53] alter complete (version 2)
I20240503 01:59:42.094902 10953 raft_consensus.cc:2932] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:42.097106 10668 consensus_queue.cc:1035] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:42.101258 7012 raft_consensus.cc:2932] T 8ed2793842464443826f83f60fb6dcd1 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:42.103521 6735 catalog_manager.cc:5484] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:42.108548 9494 raft_consensus.cc:2932] T 8ed2793842464443826f83f60fb6dcd1 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:42.108722 11520 consensus_queue.cc:1035] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:42.115005 6863 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ed2793842464443826f83f60fb6dcd1 with delete_type TABLET_DATA_TOMBSTONED (TS 5dcc33cc621a4cbbae155f3230fb7c46 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:42.115278 11726 tablet_replica.cc:330] T 8ed2793842464443826f83f60fb6dcd1 P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:42.115443 11726 raft_consensus.cc:2229] T 8ed2793842464443826f83f60fb6dcd1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:42.115610 11726 raft_consensus.cc:2258] T 8ed2793842464443826f83f60fb6dcd1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:42.116317 11726 ts_tablet_manager.cc:1875] T 8ed2793842464443826f83f60fb6dcd1 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:42.118276 11726 ts_tablet_manager.cc:1888] T 8ed2793842464443826f83f60fb6dcd1 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 01:59:42.118348 11726 log.cc:1196] T 8ed2793842464443826f83f60fb6dcd1 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/8ed2793842464443826f83f60fb6dcd1
I20240503 01:59:42.119508 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 8ed2793842464443826f83f60fb6dcd1 (table rebalance_test_table_3 [id=45e94c90bda847688db4572a4f9d0c02]) successfully deleted
I20240503 01:59:42.203703 9341 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:42.206230 7256 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:42.209157 7124 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:42.210836 6736 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:42.211257 9473 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:42.217121 9605 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:42.220327 11766 tablet.cc:1722] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:42.221351 11776 tablet.cc:1722] T a080239d59ae4fd1be9bb799871130f1 P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:42.223407 11767 tablet.cc:1722] T a080239d59ae4fd1be9bb799871130f1 P 1d0e738c34af40fa9950c6f086619b3a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:42.227087 6738 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=7bc73d615e1b43e39c08cc44a0cbac53] alter complete (version 3)
I20240503 01:59:42.240123 6992 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:42.246860 6736 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:42.255153 9736 tablet_service.cc:1467] Processing CreateTablet for tablet b3ce9de18bda410aa204bf7c1c32e1b7 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=607fa4fc037b4c5ba04342cb6d7fd017]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:42.255254 9604 tablet_service.cc:1467] Processing CreateTablet for tablet 2755530dc6f04baaaafd5d5214b150af (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=607fa4fc037b4c5ba04342cb6d7fd017]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:42.255522 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3ce9de18bda410aa204bf7c1c32e1b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:42.255533 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2755530dc6f04baaaafd5d5214b150af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:42.255504 7256 tablet_service.cc:1467] Processing CreateTablet for tablet aa71e80f211f4cce97188f6ac8fd3e8c (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=607fa4fc037b4c5ba04342cb6d7fd017]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:42.255759 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa71e80f211f4cce97188f6ac8fd3e8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:42.257934 11613 tablet_bootstrap.cc:492] T 2755530dc6f04baaaafd5d5214b150af P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:42.258041 11700 tablet_bootstrap.cc:492] T aa71e80f211f4cce97188f6ac8fd3e8c P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:42.258968 11700 tablet_bootstrap.cc:654] T aa71e80f211f4cce97188f6ac8fd3e8c P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:42.258994 11613 tablet_bootstrap.cc:654] T 2755530dc6f04baaaafd5d5214b150af P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:42.259409 9473 tablet_service.cc:1467] Processing CreateTablet for tablet b3ce9de18bda410aa204bf7c1c32e1b7 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=607fa4fc037b4c5ba04342cb6d7fd017]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:42.259689 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3ce9de18bda410aa204bf7c1c32e1b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:42.260064 11700 tablet_bootstrap.cc:492] T aa71e80f211f4cce97188f6ac8fd3e8c P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:42.260145 11700 ts_tablet_manager.cc:1367] T aa71e80f211f4cce97188f6ac8fd3e8c P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:42.260121 9341 tablet_service.cc:1467] Processing CreateTablet for tablet 2755530dc6f04baaaafd5d5214b150af (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=607fa4fc037b4c5ba04342cb6d7fd017]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:42.260250 6992 tablet_service.cc:1467] Processing CreateTablet for tablet b3ce9de18bda410aa204bf7c1c32e1b7 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=607fa4fc037b4c5ba04342cb6d7fd017]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:42.260375 9341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2755530dc6f04baaaafd5d5214b150af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:42.260493 6992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3ce9de18bda410aa204bf7c1c32e1b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:42.260593 11700 raft_consensus.cc:348] T aa71e80f211f4cce97188f6ac8fd3e8c P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:42.260735 11700 raft_consensus.cc:374] T aa71e80f211f4cce97188f6ac8fd3e8c P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:42.260766 11700 raft_consensus.cc:729] T aa71e80f211f4cce97188f6ac8fd3e8c P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:42.260880 11700 consensus_queue.cc:260] T aa71e80f211f4cce97188f6ac8fd3e8c P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:42.261133 11700 ts_tablet_manager.cc:1398] T aa71e80f211f4cce97188f6ac8fd3e8c P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 01:59:42.261497 6993 tablet_service.cc:1467] Processing CreateTablet for tablet aa71e80f211f4cce97188f6ac8fd3e8c (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=607fa4fc037b4c5ba04342cb6d7fd017]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:42.261730 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa71e80f211f4cce97188f6ac8fd3e8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:42.262861 6994 tablet_service.cc:1467] Processing CreateTablet for tablet 2755530dc6f04baaaafd5d5214b150af (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=607fa4fc037b4c5ba04342cb6d7fd017]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:42.263095 6994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2755530dc6f04baaaafd5d5214b150af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:42.255254 9605 tablet_service.cc:1467] Processing CreateTablet for tablet aa71e80f211f4cce97188f6ac8fd3e8c (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=607fa4fc037b4c5ba04342cb6d7fd017]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:42.265219 9605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa71e80f211f4cce97188f6ac8fd3e8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:42.266911 11860 tablet_bootstrap.cc:492] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:42.267443 11863 tablet_bootstrap.cc:492] T b3ce9de18bda410aa204bf7c1c32e1b7 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:42.269094 11860 tablet_bootstrap.cc:654] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:42.269601 11613 tablet_bootstrap.cc:492] T 2755530dc6f04baaaafd5d5214b150af P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:42.269703 11613 ts_tablet_manager.cc:1367] T 2755530dc6f04baaaafd5d5214b150af P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 01:59:42.270464 11613 raft_consensus.cc:348] T 2755530dc6f04baaaafd5d5214b150af P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:42.270606 11613 raft_consensus.cc:374] T 2755530dc6f04baaaafd5d5214b150af P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:42.270643 11613 raft_consensus.cc:729] T 2755530dc6f04baaaafd5d5214b150af P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:42.270771 11613 consensus_queue.cc:260] T 2755530dc6f04baaaafd5d5214b150af P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:42.271790 11865 tablet_bootstrap.cc:492] T b3ce9de18bda410aa204bf7c1c32e1b7 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:42.272678 11865 tablet_bootstrap.cc:654] T b3ce9de18bda410aa204bf7c1c32e1b7 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:42.273032 11864 tablet_bootstrap.cc:492] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:42.273941 11864 tablet_bootstrap.cc:654] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:42.275202 11864 tablet_bootstrap.cc:492] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:42.275290 11864 ts_tablet_manager.cc:1367] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:42.275620 11865 tablet_bootstrap.cc:492] T b3ce9de18bda410aa204bf7c1c32e1b7 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:42.275700 11865 ts_tablet_manager.cc:1367] T b3ce9de18bda410aa204bf7c1c32e1b7 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:42.275830 11864 raft_consensus.cc:348] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:42.275947 11864 raft_consensus.cc:374] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:42.275990 11864 raft_consensus.cc:729] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:42.276114 11864 consensus_queue.cc:260] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:42.276098 11865 raft_consensus.cc:348] T b3ce9de18bda410aa204bf7c1c32e1b7 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:42.276252 11865 raft_consensus.cc:374] T b3ce9de18bda410aa204bf7c1c32e1b7 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:42.276279 11865 raft_consensus.cc:729] T b3ce9de18bda410aa204bf7c1c32e1b7 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:42.276387 11864 ts_tablet_manager.cc:1398] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:42.276391 11865 consensus_queue.cc:260] T b3ce9de18bda410aa204bf7c1c32e1b7 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:42.279119 11865 ts_tablet_manager.cc:1398] T b3ce9de18bda410aa204bf7c1c32e1b7 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:42.279435 11865 tablet_bootstrap.cc:492] T 2755530dc6f04baaaafd5d5214b150af P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:42.279294 6863 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:42.279881 11613 ts_tablet_manager.cc:1398] T 2755530dc6f04baaaafd5d5214b150af P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 01:59:42.280278 11613 tablet_bootstrap.cc:492] T aa71e80f211f4cce97188f6ac8fd3e8c P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:42.282228 11613 tablet_bootstrap.cc:654] T aa71e80f211f4cce97188f6ac8fd3e8c P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:42.284279 11863 tablet_bootstrap.cc:654] T b3ce9de18bda410aa204bf7c1c32e1b7 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:42.276882 9736 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:42.294014 11865 tablet_bootstrap.cc:654] T 2755530dc6f04baaaafd5d5214b150af P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:42.294346 11613 tablet_bootstrap.cc:492] T aa71e80f211f4cce97188f6ac8fd3e8c P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:42.294543 11613 ts_tablet_manager.cc:1367] T aa71e80f211f4cce97188f6ac8fd3e8c P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240503 01:59:42.295148 11613 raft_consensus.cc:348] T aa71e80f211f4cce97188f6ac8fd3e8c P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:42.295349 11613 raft_consensus.cc:374] T aa71e80f211f4cce97188f6ac8fd3e8c P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:42.295433 11613 raft_consensus.cc:729] T aa71e80f211f4cce97188f6ac8fd3e8c P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:42.295662 11613 consensus_queue.cc:260] T aa71e80f211f4cce97188f6ac8fd3e8c P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:42.296846 11613 ts_tablet_manager.cc:1398] T aa71e80f211f4cce97188f6ac8fd3e8c P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:42.297767 11860 tablet_bootstrap.cc:492] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:42.297920 11860 ts_tablet_manager.cc:1367] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.000s
I20240503 01:59:42.299031 11860 raft_consensus.cc:348] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:42.299239 11860 raft_consensus.cc:374] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:42.299309 11860 raft_consensus.cc:729] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:42.299484 11860 consensus_queue.cc:260] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:42.299834 11860 ts_tablet_manager.cc:1398] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:42.301446 11863 tablet_bootstrap.cc:492] T b3ce9de18bda410aa204bf7c1c32e1b7 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:42.301609 11863 ts_tablet_manager.cc:1367] T b3ce9de18bda410aa204bf7c1c32e1b7 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.000s
I20240503 01:59:42.302117 11863 raft_consensus.cc:348] T b3ce9de18bda410aa204bf7c1c32e1b7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:42.302289 11863 raft_consensus.cc:374] T b3ce9de18bda410aa204bf7c1c32e1b7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:42.302353 11863 raft_consensus.cc:729] T b3ce9de18bda410aa204bf7c1c32e1b7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:42.302568 11863 consensus_queue.cc:260] T b3ce9de18bda410aa204bf7c1c32e1b7 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:42.303501 11863 ts_tablet_manager.cc:1398] T b3ce9de18bda410aa204bf7c1c32e1b7 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:42.315099 11865 tablet_bootstrap.cc:492] T 2755530dc6f04baaaafd5d5214b150af P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:42.315230 11865 ts_tablet_manager.cc:1367] T 2755530dc6f04baaaafd5d5214b150af P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.000s
I20240503 01:59:42.315755 11865 raft_consensus.cc:348] T 2755530dc6f04baaaafd5d5214b150af P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:42.315876 11865 raft_consensus.cc:374] T 2755530dc6f04baaaafd5d5214b150af P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:42.315912 11865 raft_consensus.cc:729] T 2755530dc6f04baaaafd5d5214b150af P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:42.316071 11865 consensus_queue.cc:260] T 2755530dc6f04baaaafd5d5214b150af P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:42.316457 11865 ts_tablet_manager.cc:1398] T 2755530dc6f04baaaafd5d5214b150af P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:42.316913 11865 tablet_bootstrap.cc:492] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:42.317852 11865 tablet_bootstrap.cc:654] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:42.319598 11865 tablet_bootstrap.cc:492] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:42.319697 11865 ts_tablet_manager.cc:1367] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:42.320188 11865 raft_consensus.cc:348] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:42.320345 11865 raft_consensus.cc:374] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:42.320406 11865 raft_consensus.cc:729] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:42.320549 11865 consensus_queue.cc:260] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:42.320981 11865 ts_tablet_manager.cc:1398] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:42.336566 6734 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:42.336740 6734 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:42.339541 6734 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet a080239d59ae4fd1be9bb799871130f1
I20240503 01:59:42.340188 7256 tablet_service.cc:1514] Processing DeleteTablet for tablet a080239d59ae4fd1be9bb799871130f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:42 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:42.340406 9605 tablet_service.cc:1514] Processing DeleteTablet for tablet a080239d59ae4fd1be9bb799871130f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:42 UTC) from {username='slave'} at 127.0.0.1:58562
I20240503 01:59:42.340435 6863 tablet_service.cc:1514] Processing DeleteTablet for tablet a080239d59ae4fd1be9bb799871130f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:42 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:42.340521 11878 tablet_replica.cc:330] T a080239d59ae4fd1be9bb799871130f1 P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:42.340632 11726 tablet_replica.cc:330] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:42.340682 11878 raft_consensus.cc:2229] T a080239d59ae4fd1be9bb799871130f1 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:42.340766 11726 raft_consensus.cc:2229] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:42.340798 11878 raft_consensus.cc:2258] T a080239d59ae4fd1be9bb799871130f1 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:42.340988 11726 raft_consensus.cc:2258] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:42.341849 11726 ts_tablet_manager.cc:1875] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:42.342974 11879 tablet_replica.cc:330] T a080239d59ae4fd1be9bb799871130f1 P b7062fbbc2b54fff9e99ec4ea5562c76: stopping tablet replica
I20240503 01:59:42.343289 11879 raft_consensus.cc:2229] T a080239d59ae4fd1be9bb799871130f1 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:42.343658 11879 raft_consensus.cc:2258] T a080239d59ae4fd1be9bb799871130f1 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:42.344223 11726 ts_tablet_manager.cc:1888] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:42.344377 11879 ts_tablet_manager.cc:1875] T a080239d59ae4fd1be9bb799871130f1 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:42.344400 11726 log.cc:1196] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/a080239d59ae4fd1be9bb799871130f1
I20240503 01:59:42.344708 11726 ts_tablet_manager.cc:1909] T a080239d59ae4fd1be9bb799871130f1 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:42.346175 11878 ts_tablet_manager.cc:1875] T a080239d59ae4fd1be9bb799871130f1 P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:42.346650 11879 ts_tablet_manager.cc:1888] T a080239d59ae4fd1be9bb799871130f1 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:42.346787 11879 log.cc:1196] T a080239d59ae4fd1be9bb799871130f1 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/a080239d59ae4fd1be9bb799871130f1
I20240503 01:59:42.347133 11879 ts_tablet_manager.cc:1909] T a080239d59ae4fd1be9bb799871130f1 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting consensus metadata
I20240503 01:59:42.347445 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet a080239d59ae4fd1be9bb799871130f1 (table rebalancer_dynamic_table_new_name [id=7bc73d615e1b43e39c08cc44a0cbac53]) successfully deleted
I20240503 01:59:42.348050 6727 catalog_manager.cc:4830] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): tablet a080239d59ae4fd1be9bb799871130f1 (table rebalancer_dynamic_table_new_name [id=7bc73d615e1b43e39c08cc44a0cbac53]) successfully deleted
I20240503 01:59:42.348963 11878 ts_tablet_manager.cc:1888] T a080239d59ae4fd1be9bb799871130f1 P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:42.349645 11878 log.cc:1196] T a080239d59ae4fd1be9bb799871130f1 P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/a080239d59ae4fd1be9bb799871130f1
I20240503 01:59:42.350006 11878 ts_tablet_manager.cc:1909] T a080239d59ae4fd1be9bb799871130f1 P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:42.350705 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet a080239d59ae4fd1be9bb799871130f1 (table rebalancer_dynamic_table_new_name [id=7bc73d615e1b43e39c08cc44a0cbac53]) successfully deleted
I20240503 01:59:42.367626 11786 raft_consensus.cc:482] T aa71e80f211f4cce97188f6ac8fd3e8c P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:42.367671 9757 consensus_queue.cc:237] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:42.367749 11786 raft_consensus.cc:504] T aa71e80f211f4cce97188f6ac8fd3e8c P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:42.368235 11786 leader_election.cc:288] T aa71e80f211f4cce97188f6ac8fd3e8c P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:42.368847 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa71e80f211f4cce97188f6ac8fd3e8c" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:42.368839 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa71e80f211f4cce97188f6ac8fd3e8c" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:42.368970 7012 raft_consensus.cc:2454] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 01:59:42.368974 7275 raft_consensus.cc:2454] T aa71e80f211f4cce97188f6ac8fd3e8c P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 01:59:42.369022 6882 raft_consensus.cc:1261] T 31b077072738410f8d7c139ad1b5aa9c P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:42.369272 9559 leader_election.cc:302] T aa71e80f211f4cce97188f6ac8fd3e8c P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:42.369431 11786 raft_consensus.cc:2781] T aa71e80f211f4cce97188f6ac8fd3e8c P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:42.369469 11786 raft_consensus.cc:482] T aa71e80f211f4cce97188f6ac8fd3e8c P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:42.369417 11127 consensus_queue.cc:1035] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:42.369494 11786 raft_consensus.cc:3037] T aa71e80f211f4cce97188f6ac8fd3e8c P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:42.369479 9363 raft_consensus.cc:1261] T 31b077072738410f8d7c139ad1b5aa9c P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:42.370463 11127 consensus_queue.cc:1035] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:42.370558 11786 raft_consensus.cc:504] T aa71e80f211f4cce97188f6ac8fd3e8c P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:42.370906 11786 leader_election.cc:288] T aa71e80f211f4cce97188f6ac8fd3e8c P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:42.371593 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa71e80f211f4cce97188f6ac8fd3e8c" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:42.371719 7275 raft_consensus.cc:3037] T aa71e80f211f4cce97188f6ac8fd3e8c P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:42.371626 11127 raft_consensus.cc:2932] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:42.372869 7275 raft_consensus.cc:2454] T aa71e80f211f4cce97188f6ac8fd3e8c P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 01:59:42.373046 9363 raft_consensus.cc:2932] T 31b077072738410f8d7c139ad1b5aa9c P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:42.373381 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa71e80f211f4cce97188f6ac8fd3e8c" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:42.373505 7012 raft_consensus.cc:3037] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:42.373708 6882 raft_consensus.cc:2932] T 31b077072738410f8d7c139ad1b5aa9c P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:42.374496 7012 raft_consensus.cc:2454] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 01:59:42.375172 9560 leader_election.cc:302] T aa71e80f211f4cce97188f6ac8fd3e8c P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:42.375910 6734 catalog_manager.cc:5484] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:42.378269 11786 raft_consensus.cc:2781] T aa71e80f211f4cce97188f6ac8fd3e8c P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:42.378377 11786 raft_consensus.cc:686] T aa71e80f211f4cce97188f6ac8fd3e8c P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 LEADER]: Becoming Leader. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Running, Role: LEADER
I20240503 01:59:42.378553 11786 consensus_queue.cc:237] T aa71e80f211f4cce97188f6ac8fd3e8c P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:42.380628 6738 catalog_manager.cc:5484] T aa71e80f211f4cce97188f6ac8fd3e8c P b7062fbbc2b54fff9e99ec4ea5562c76 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:42.381794 9473 tablet_service.cc:1514] Processing DeleteTablet for tablet 31b077072738410f8d7c139ad1b5aa9c with delete_type TABLET_DATA_TOMBSTONED (TS bb5364d730cb4489a0bfc6c270b7450a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54678
I20240503 01:59:42.382227 11900 tablet_replica.cc:330] T 31b077072738410f8d7c139ad1b5aa9c P bb5364d730cb4489a0bfc6c270b7450a: stopping tablet replica
I20240503 01:59:42.382441 11900 raft_consensus.cc:2229] T 31b077072738410f8d7c139ad1b5aa9c P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:42.382711 11900 raft_consensus.cc:2258] T 31b077072738410f8d7c139ad1b5aa9c P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:42.383239 11900 ts_tablet_manager.cc:1875] T 31b077072738410f8d7c139ad1b5aa9c P bb5364d730cb4489a0bfc6c270b7450a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:42.385775 11900 ts_tablet_manager.cc:1888] T 31b077072738410f8d7c139ad1b5aa9c P bb5364d730cb4489a0bfc6c270b7450a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 01:59:42.385851 11900 log.cc:1196] T 31b077072738410f8d7c139ad1b5aa9c P bb5364d730cb4489a0bfc6c270b7450a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/31b077072738410f8d7c139ad1b5aa9c
I20240503 01:59:42.386487 6725 catalog_manager.cc:4830] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): tablet 31b077072738410f8d7c139ad1b5aa9c (table rebalancer_extra_table_0 [id=72ea0c008b3e43ccab2d5b224cf8896b]) successfully deleted
I20240503 01:59:42.407058 11326 raft_consensus.cc:482] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:42.407209 11326 raft_consensus.cc:504] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:42.407635 11326 leader_election.cc:288] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:42.408172 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ce9de18bda410aa204bf7c1c32e1b7" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:42.408404 9494 raft_consensus.cc:2454] T b3ce9de18bda410aa204bf7c1c32e1b7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 01:59:42.408771 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ce9de18bda410aa204bf7c1c32e1b7" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:42.408828 9690 leader_election.cc:302] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:42.408969 7012 raft_consensus.cc:2454] T b3ce9de18bda410aa204bf7c1c32e1b7 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 01:59:42.409065 11326 raft_consensus.cc:2781] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:42.409130 11326 raft_consensus.cc:482] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:42.409183 11326 raft_consensus.cc:3037] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:42.411087 11326 raft_consensus.cc:504] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:42.411738 11326 leader_election.cc:288] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Requested vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:42.412178 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ce9de18bda410aa204bf7c1c32e1b7" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:42.412216 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ce9de18bda410aa204bf7c1c32e1b7" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:42.412339 7012 raft_consensus.cc:3037] T b3ce9de18bda410aa204bf7c1c32e1b7 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:42.412354 9494 raft_consensus.cc:3037] T b3ce9de18bda410aa204bf7c1c32e1b7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:42.413216 7012 raft_consensus.cc:2454] T b3ce9de18bda410aa204bf7c1c32e1b7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:42.413408 9494 raft_consensus.cc:2454] T b3ce9de18bda410aa204bf7c1c32e1b7 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:42.413573 9690 leader_election.cc:302] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:42.413913 11326 raft_consensus.cc:2781] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:42.414191 11326 raft_consensus.cc:686] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Becoming Leader. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Running, Role: LEADER
I20240503 01:59:42.414389 11326 consensus_queue.cc:237] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:42.417157 6738 catalog_manager.cc:5484] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: term changed from 0 to 1, leader changed from <none> to db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:42.441346 6739 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:42.447639 9473 tablet_service.cc:1467] Processing CreateTablet for tablet adfc7405d029466ab16fdd8955554f76 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=76d0e56c39964d109a70c4b51a82da1d]), partition=
I20240503 01:59:42.447801 9736 tablet_service.cc:1467] Processing CreateTablet for tablet adfc7405d029466ab16fdd8955554f76 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=76d0e56c39964d109a70c4b51a82da1d]), partition=
I20240503 01:59:42.447834 7256 tablet_service.cc:1467] Processing CreateTablet for tablet adfc7405d029466ab16fdd8955554f76 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=76d0e56c39964d109a70c4b51a82da1d]), partition=
I20240503 01:59:42.447933 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adfc7405d029466ab16fdd8955554f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:42.448020 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adfc7405d029466ab16fdd8955554f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:42.448117 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adfc7405d029466ab16fdd8955554f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:42.450207 11860 tablet_bootstrap.cc:492] T adfc7405d029466ab16fdd8955554f76 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:42.450354 11863 tablet_bootstrap.cc:492] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:42.450598 11700 tablet_bootstrap.cc:492] T adfc7405d029466ab16fdd8955554f76 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:42.451270 11860 tablet_bootstrap.cc:654] T adfc7405d029466ab16fdd8955554f76 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:42.451270 11863 tablet_bootstrap.cc:654] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:42.451784 11700 tablet_bootstrap.cc:654] T adfc7405d029466ab16fdd8955554f76 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:42.452885 11863 tablet_bootstrap.cc:492] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:42.452980 11863 ts_tablet_manager.cc:1367] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:42.453521 11863 raft_consensus.cc:348] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:42.453639 11863 raft_consensus.cc:374] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:42.453676 11863 raft_consensus.cc:729] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:42.453795 11863 consensus_queue.cc:260] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:42.453892 11527 raft_consensus.cc:482] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:42.454020 11527 raft_consensus.cc:504] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:42.454140 11863 ts_tablet_manager.cc:1398] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:42.454424 11527 leader_election.cc:288] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:42.455088 11700 tablet_bootstrap.cc:492] T adfc7405d029466ab16fdd8955554f76 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:42.455070 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2755530dc6f04baaaafd5d5214b150af" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:42.455174 11700 ts_tablet_manager.cc:1367] T adfc7405d029466ab16fdd8955554f76 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:42.455215 7012 raft_consensus.cc:2454] T 2755530dc6f04baaaafd5d5214b150af P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:42.455509 9296 leader_election.cc:302] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:42.455713 11700 raft_consensus.cc:348] T adfc7405d029466ab16fdd8955554f76 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:42.455812 11700 raft_consensus.cc:374] T adfc7405d029466ab16fdd8955554f76 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:42.455845 11700 raft_consensus.cc:729] T adfc7405d029466ab16fdd8955554f76 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:42.455935 9626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2755530dc6f04baaaafd5d5214b150af" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:42.455962 11700 consensus_queue.cc:260] T adfc7405d029466ab16fdd8955554f76 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:42.456082 9626 raft_consensus.cc:2454] T 2755530dc6f04baaaafd5d5214b150af P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:42.456274 11700 ts_tablet_manager.cc:1398] T adfc7405d029466ab16fdd8955554f76 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:42.456526 11527 raft_consensus.cc:2781] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:42.456581 11527 raft_consensus.cc:482] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:42.456624 11527 raft_consensus.cc:3037] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:42.457731 11527 raft_consensus.cc:504] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:42.458106 11527 leader_election.cc:288] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:42.458645 11860 tablet_bootstrap.cc:492] T adfc7405d029466ab16fdd8955554f76 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:42.458626 9626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2755530dc6f04baaaafd5d5214b150af" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:42.458751 9626 raft_consensus.cc:3037] T 2755530dc6f04baaaafd5d5214b150af P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:42.458760 11860 ts_tablet_manager.cc:1367] T adfc7405d029466ab16fdd8955554f76 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.001s
I20240503 01:59:42.459367 11860 raft_consensus.cc:348] T adfc7405d029466ab16fdd8955554f76 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:42.459477 11860 raft_consensus.cc:374] T adfc7405d029466ab16fdd8955554f76 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:42.459511 11860 raft_consensus.cc:729] T adfc7405d029466ab16fdd8955554f76 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:42.459628 11860 consensus_queue.cc:260] T adfc7405d029466ab16fdd8955554f76 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:42.459842 9626 raft_consensus.cc:2454] T 2755530dc6f04baaaafd5d5214b150af P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:42.459931 11860 ts_tablet_manager.cc:1398] T adfc7405d029466ab16fdd8955554f76 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:42.460124 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2755530dc6f04baaaafd5d5214b150af" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:42.460253 7012 raft_consensus.cc:3037] T 2755530dc6f04baaaafd5d5214b150af P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:42.461318 7012 raft_consensus.cc:2454] T 2755530dc6f04baaaafd5d5214b150af P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:42.462405 9296 leader_election.cc:302] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:42.467149 11527 raft_consensus.cc:2781] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:42.467377 11527 raft_consensus.cc:686] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Running, Role: LEADER
I20240503 01:59:42.467559 11527 consensus_queue.cc:237] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:42.469446 6739 catalog_manager.cc:5484] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:42.503270 9341 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:42.503258 7256 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:42.511023 6863 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:42.516424 6993 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:42.519896 10359 raft_consensus.cc:482] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:42.520018 10359 raft_consensus.cc:504] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:42.520391 10359 leader_election.cc:288] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:42.521047 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adfc7405d029466ab16fdd8955554f76" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:42.521178 9757 raft_consensus.cc:2454] T adfc7405d029466ab16fdd8955554f76 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:42.521520 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adfc7405d029466ab16fdd8955554f76" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:42.521636 7275 raft_consensus.cc:2454] T adfc7405d029466ab16fdd8955554f76 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:42.522331 9428 leader_election.cc:302] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:42.522564 10359 raft_consensus.cc:2781] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:42.522627 10359 raft_consensus.cc:482] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:42.522666 10359 raft_consensus.cc:3037] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:42.523725 10359 raft_consensus.cc:504] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:42.524114 10359 leader_election.cc:288] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:42.525030 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adfc7405d029466ab16fdd8955554f76" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:42.525044 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adfc7405d029466ab16fdd8955554f76" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:42.525177 7275 raft_consensus.cc:3037] T adfc7405d029466ab16fdd8955554f76 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:42.525174 9757 raft_consensus.cc:3037] T adfc7405d029466ab16fdd8955554f76 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:42.526139 7275 raft_consensus.cc:2454] T adfc7405d029466ab16fdd8955554f76 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:42.526221 9757 raft_consensus.cc:2454] T adfc7405d029466ab16fdd8955554f76 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:42.526728 9428 leader_election.cc:302] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:42.526973 10359 raft_consensus.cc:2781] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:42.527115 10359 raft_consensus.cc:686] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Becoming Leader. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Running, Role: LEADER
I20240503 01:59:42.527252 10359 consensus_queue.cc:237] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:42.529628 6739 catalog_manager.cc:5484] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: term changed from 0 to 1, leader changed from <none> to bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:42.533529 9736 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:42.538847 7124 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:42.541316 9473 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:42.542706 9605 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:42.631924 6882 consensus_queue.cc:237] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:42.633522 7275 raft_consensus.cc:1261] T 72e090a6c18e4a13bbb343eb40a07694 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:42.633527 7143 raft_consensus.cc:1261] T 72e090a6c18e4a13bbb343eb40a07694 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:42.633627 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72e090a6c18e4a13bbb343eb40a07694. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:42.633993 10668 consensus_queue.cc:1035] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:42.634198 11859 consensus_queue.cc:1035] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:42.636359 11859 raft_consensus.cc:2932] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:42.636981 7143 raft_consensus.cc:2932] T 72e090a6c18e4a13bbb343eb40a07694 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:42.639237 7275 raft_consensus.cc:2932] T 72e090a6c18e4a13bbb343eb40a07694 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:42.639573 6739 catalog_manager.cc:5484] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:42.655910 6735 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:42.660329 9757 raft_consensus.cc:1261] T adfc7405d029466ab16fdd8955554f76 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:42.660435 7275 raft_consensus.cc:1261] T adfc7405d029466ab16fdd8955554f76 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:42.660912 10359 consensus_queue.cc:1035] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:42.661211 11947 consensus_queue.cc:1035] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:42.664261 6738 catalog_manager.cc:5576] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271) does not have the latest schema for tablet adfc7405d029466ab16fdd8955554f76 (table rebalancer_dynamic_table [id=76d0e56c39964d109a70c4b51a82da1d]). Expected version 1 got 0
I20240503 01:59:42.664685 6735 catalog_manager.cc:5576] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053) does not have the latest schema for tablet adfc7405d029466ab16fdd8955554f76 (table rebalancer_dynamic_table [id=76d0e56c39964d109a70c4b51a82da1d]). Expected version 1 got 0
I20240503 01:59:42.665059 11986 tablet.cc:1722] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:42.665488 11986 tablet.cc:1620] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:42.667343 6735 catalog_manager.cc:5576] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053) does not have the latest schema for tablet adfc7405d029466ab16fdd8955554f76 (table rebalancer_dynamic_table [id=76d0e56c39964d109a70c4b51a82da1d]). Expected version 1 got 0
I20240503 01:59:42.668413 11987 tablet.cc:1722] T adfc7405d029466ab16fdd8955554f76 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:42.669659 11767 tablet.cc:1722] T adfc7405d029466ab16fdd8955554f76 P 1d0e738c34af40fa9950c6f086619b3a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:42.669989 11767 tablet.cc:1620] T adfc7405d029466ab16fdd8955554f76 P 1d0e738c34af40fa9950c6f086619b3a: MemRowSet was empty: no flush needed.
I20240503 01:59:42.670228 11987 tablet.cc:1620] T adfc7405d029466ab16fdd8955554f76 P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:42.671593 6738 catalog_manager.cc:5576] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271) does not have the latest schema for tablet adfc7405d029466ab16fdd8955554f76 (table rebalancer_dynamic_table [id=76d0e56c39964d109a70c4b51a82da1d]). Expected version 1 got 0
I20240503 01:59:42.671810 6735 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=76d0e56c39964d109a70c4b51a82da1d] alter complete (version 1)
I20240503 01:59:42.674645 7275 consensus_queue.cc:237] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:42.676246 7143 raft_consensus.cc:1261] T d2fbadb911aa4bd7b4db021860c202d3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:42.676399 7012 raft_consensus.cc:1261] T d2fbadb911aa4bd7b4db021860c202d3 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:42.676672 11989 consensus_queue.cc:1035] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:42.676872 11114 consensus_queue.cc:1035] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:42.677075 7210 consensus_peers.cc:485] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a -> Peer db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): Couldn't send request to peer db1a0000aaa346f79c0aa4b6c5105191. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d2fbadb911aa4bd7b4db021860c202d3. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:42.678989 11114 raft_consensus.cc:2932] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:42.679488 7012 raft_consensus.cc:2932] T d2fbadb911aa4bd7b4db021860c202d3 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:42.681686 7143 raft_consensus.cc:2932] T d2fbadb911aa4bd7b4db021860c202d3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:42.682111 6735 catalog_manager.cc:5484] T d2fbadb911aa4bd7b4db021860c202d3 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:42.707325 9757 consensus_queue.cc:237] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:42.709430 7143 raft_consensus.cc:1261] T 2f271b67a6f34eb18c98824ca4b26d17 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:42.709566 9626 raft_consensus.cc:1261] T 2f271b67a6f34eb18c98824ca4b26d17 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:42.709887 11784 consensus_queue.cc:1035] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 01:59:42.710183 9691 consensus_peers.cc:485] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): Couldn't send request to peer 1d0e738c34af40fa9950c6f086619b3a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f271b67a6f34eb18c98824ca4b26d17. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:42.710179 11326 consensus_queue.cc:1035] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:42.712461 11784 raft_consensus.cc:2932] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:42.712925 7143 raft_consensus.cc:2932] T 2f271b67a6f34eb18c98824ca4b26d17 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:42.715654 9626 raft_consensus.cc:2932] T 2f271b67a6f34eb18c98824ca4b26d17 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:42.716023 6738 catalog_manager.cc:5484] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:42.738420 7143 consensus_queue.cc:237] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:42.740427 7275 raft_consensus.cc:1261] T 76e8697ac8624e03a46bddae35f4c9fa P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:42.740670 7078 consensus_peers.cc:485] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 76e8697ac8624e03a46bddae35f4c9fa. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:42.740788 7012 raft_consensus.cc:1261] T 76e8697ac8624e03a46bddae35f4c9fa P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:42.741092 12028 consensus_queue.cc:1035] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:42.741459 11812 consensus_queue.cc:1035] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:42.743592 12028 raft_consensus.cc:2932] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:42.744009 7012 raft_consensus.cc:2932] T 76e8697ac8624e03a46bddae35f4c9fa P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:42.747809 6738 catalog_manager.cc:5484] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:42.748129 7274 raft_consensus.cc:2932] T 76e8697ac8624e03a46bddae35f4c9fa P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:42.775521 9626 consensus_queue.cc:237] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:42.776741 6739 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:42.777372 9363 raft_consensus.cc:1261] T 1d182ce2c5084082a0990ce8090eb88b P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer b7062fbbc2b54fff9e99ec4ea5562c76: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:42.777801 11786 consensus_queue.cc:1035] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:42.777968 9757 raft_consensus.cc:1261] T 1d182ce2c5084082a0990ce8090eb88b P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer b7062fbbc2b54fff9e99ec4ea5562c76: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:42.778334 11786 consensus_queue.cc:1035] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 01:59:42.779026 9560 consensus_peers.cc:485] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76 -> Peer 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): Couldn't send request to peer 1d0e738c34af40fa9950c6f086619b3a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d182ce2c5084082a0990ce8090eb88b. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:42.781311 11786 raft_consensus.cc:2932] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:42.785363 9755 raft_consensus.cc:2932] T 1d182ce2c5084082a0990ce8090eb88b P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:42.786638 9362 raft_consensus.cc:2932] T 1d182ce2c5084082a0990ce8090eb88b P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:42.789844 11986 tablet.cc:1722] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:42.790318 11986 tablet.cc:1620] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:42.791244 6738 catalog_manager.cc:5484] T 1d182ce2c5084082a0990ce8090eb88b P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:42.792676 11767 tablet.cc:1722] T adfc7405d029466ab16fdd8955554f76 P 1d0e738c34af40fa9950c6f086619b3a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:42.793045 11767 tablet.cc:1620] T adfc7405d029466ab16fdd8955554f76 P 1d0e738c34af40fa9950c6f086619b3a: MemRowSet was empty: no flush needed.
I20240503 01:59:42.793172 11987 tablet.cc:1722] T adfc7405d029466ab16fdd8955554f76 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:42.793463 11987 tablet.cc:1620] T adfc7405d029466ab16fdd8955554f76 P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:42.796744 6734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=76d0e56c39964d109a70c4b51a82da1d] alter complete (version 2)
I20240503 01:59:42.821278 9494 consensus_queue.cc:237] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:42.822921 9755 raft_consensus.cc:1261] T 31f5289f1dff4fcaa3492c16e40a3ed7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:42.823225 9626 raft_consensus.cc:1261] T 31f5289f1dff4fcaa3492c16e40a3ed7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:42.823446 10359 consensus_queue.cc:1035] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:42.823679 12070 consensus_queue.cc:1035] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:42.825556 10359 raft_consensus.cc:2932] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:42.825979 9755 raft_consensus.cc:2932] T 31f5289f1dff4fcaa3492c16e40a3ed7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:42.828722 6735 catalog_manager.cc:5484] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:42.831579 9625 raft_consensus.cc:2932] T 31f5289f1dff4fcaa3492c16e40a3ed7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
W20240503 01:59:42.831846 9427 consensus_peers.cc:485] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a -> Peer 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): Couldn't send request to peer 63331f455be049c1bdd6c23c59ac1bef. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 31f5289f1dff4fcaa3492c16e40a3ed7. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:42.855051 11527 consensus_queue.cc:1035] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:42.859174 11527 consensus_queue.cc:1035] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:42.901407 6739 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:42.909703 11986 tablet.cc:1722] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:42.911409 11767 tablet.cc:1722] T adfc7405d029466ab16fdd8955554f76 P 1d0e738c34af40fa9950c6f086619b3a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:42.917407 11987 tablet.cc:1722] T adfc7405d029466ab16fdd8955554f76 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:42.918466 6733 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=76d0e56c39964d109a70c4b51a82da1d] alter complete (version 3)
I20240503 01:59:42.936764 6993 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:42.937822 7256 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:42.938941 9605 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:42.939539 6863 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:42.940819 9341 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:42.947177 12011 consensus_queue.cc:1035] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:42.950978 12048 consensus_queue.cc:1035] T aa71e80f211f4cce97188f6ac8fd3e8c P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:42.958722 9473 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:42.959296 7124 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:42.972409 12011 consensus_queue.cc:1035] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:42.973912 9736 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:42.976953 11786 consensus_queue.cc:1035] T aa71e80f211f4cce97188f6ac8fd3e8c P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:42.995110 6733 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:43.002966 9473 tablet_service.cc:1467] Processing CreateTablet for tablet eac1ed1eaeb04232b889f32ed1df1b40 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=a60aca1dbfe344479254879037f209f3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:43.003472 9736 tablet_service.cc:1467] Processing CreateTablet for tablet d53b8c06a6cb4ba9a6d4b20abeace85d (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=a60aca1dbfe344479254879037f209f3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:43.003605 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eac1ed1eaeb04232b889f32ed1df1b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.003795 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d53b8c06a6cb4ba9a6d4b20abeace85d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.003851 9737 tablet_service.cc:1467] Processing CreateTablet for tablet eac1ed1eaeb04232b889f32ed1df1b40 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=a60aca1dbfe344479254879037f209f3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:43.003866 9341 tablet_service.cc:1467] Processing CreateTablet for tablet d53b8c06a6cb4ba9a6d4b20abeace85d (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=a60aca1dbfe344479254879037f209f3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:43.004037 9737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eac1ed1eaeb04232b889f32ed1df1b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.004151 9341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d53b8c06a6cb4ba9a6d4b20abeace85d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.005242 9342 tablet_service.cc:1467] Processing CreateTablet for tablet aab7d8c559ff446192f471aacaa47584 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=a60aca1dbfe344479254879037f209f3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:43.005483 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aab7d8c559ff446192f471aacaa47584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.005903 7256 tablet_service.cc:1467] Processing CreateTablet for tablet aab7d8c559ff446192f471aacaa47584 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=a60aca1dbfe344479254879037f209f3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:43.006189 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aab7d8c559ff446192f471aacaa47584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.006852 9605 tablet_service.cc:1467] Processing CreateTablet for tablet d53b8c06a6cb4ba9a6d4b20abeace85d (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=a60aca1dbfe344479254879037f209f3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:43.007277 9605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d53b8c06a6cb4ba9a6d4b20abeace85d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.009445 6863 tablet_service.cc:1467] Processing CreateTablet for tablet aab7d8c559ff446192f471aacaa47584 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=a60aca1dbfe344479254879037f209f3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:43.009752 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aab7d8c559ff446192f471aacaa47584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.010609 9604 tablet_service.cc:1467] Processing CreateTablet for tablet eac1ed1eaeb04232b889f32ed1df1b40 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=a60aca1dbfe344479254879037f209f3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:43.010895 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eac1ed1eaeb04232b889f32ed1df1b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.012113 12137 tablet_bootstrap.cc:492] T aab7d8c559ff446192f471aacaa47584 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:43.012893 12137 tablet_bootstrap.cc:654] T aab7d8c559ff446192f471aacaa47584 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:43.022382 12136 tablet_bootstrap.cc:492] T d53b8c06a6cb4ba9a6d4b20abeace85d P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:43.022809 12140 tablet_bootstrap.cc:492] T eac1ed1eaeb04232b889f32ed1df1b40 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:43.023533 6735 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:43.023736 6735 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:43.024266 12140 tablet_bootstrap.cc:654] T eac1ed1eaeb04232b889f32ed1df1b40 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:43.026168 12138 tablet_bootstrap.cc:492] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:43.027346 12138 tablet_bootstrap.cc:654] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:43.028029 6735 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet adfc7405d029466ab16fdd8955554f76
I20240503 01:59:43.028414 12134 tablet_bootstrap.cc:492] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:43.028635 9473 tablet_service.cc:1514] Processing DeleteTablet for tablet adfc7405d029466ab16fdd8955554f76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:43 UTC) from {username='slave'} at 127.0.0.1:54678
I20240503 01:59:43.028999 7256 tablet_service.cc:1514] Processing DeleteTablet for tablet adfc7405d029466ab16fdd8955554f76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:43 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:43.029132 12143 tablet_replica.cc:330] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a: stopping tablet replica
I20240503 01:59:43.029564 12143 raft_consensus.cc:2229] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:43.029726 9737 tablet_service.cc:1514] Processing DeleteTablet for tablet adfc7405d029466ab16fdd8955554f76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:43 UTC) from {username='slave'} at 127.0.0.1:48152
I20240503 01:59:43.029866 12143 raft_consensus.cc:2258] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:43.030763 12134 tablet_bootstrap.cc:654] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:43.030813 12143 ts_tablet_manager.cc:1875] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:43.031581 12135 tablet_bootstrap.cc:492] T d53b8c06a6cb4ba9a6d4b20abeace85d P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:43.033054 12135 tablet_bootstrap.cc:654] T d53b8c06a6cb4ba9a6d4b20abeace85d P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:43.034633 12143 ts_tablet_manager.cc:1888] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:43.034667 12136 tablet_bootstrap.cc:654] T d53b8c06a6cb4ba9a6d4b20abeace85d P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:43.034729 12143 log.cc:1196] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/adfc7405d029466ab16fdd8955554f76
I20240503 01:59:43.035013 12143 ts_tablet_manager.cc:1909] T adfc7405d029466ab16fdd8955554f76 P bb5364d730cb4489a0bfc6c270b7450a: Deleting consensus metadata
I20240503 01:59:43.035856 6725 catalog_manager.cc:4830] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): tablet adfc7405d029466ab16fdd8955554f76 (table rebalancer_dynamic_table_new_name [id=76d0e56c39964d109a70c4b51a82da1d]) successfully deleted
I20240503 01:59:43.035960 12138 tablet_bootstrap.cc:492] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:43.036083 12138 ts_tablet_manager.cc:1367] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 01:59:43.036559 12138 raft_consensus.cc:348] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:43.036669 12138 raft_consensus.cc:374] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:43.036693 12138 raft_consensus.cc:729] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.036834 12138 consensus_queue.cc:260] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:43.037118 12138 ts_tablet_manager.cc:1398] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:43.037729 12136 tablet_bootstrap.cc:492] T d53b8c06a6cb4ba9a6d4b20abeace85d P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:43.037847 12136 ts_tablet_manager.cc:1367] T d53b8c06a6cb4ba9a6d4b20abeace85d P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 01:59:43.038332 12136 raft_consensus.cc:348] T d53b8c06a6cb4ba9a6d4b20abeace85d P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:43.038478 12136 raft_consensus.cc:374] T d53b8c06a6cb4ba9a6d4b20abeace85d P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:43.038585 12136 raft_consensus.cc:729] T d53b8c06a6cb4ba9a6d4b20abeace85d P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.038781 12136 consensus_queue.cc:260] T d53b8c06a6cb4ba9a6d4b20abeace85d P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:43.039172 12136 ts_tablet_manager.cc:1398] T d53b8c06a6cb4ba9a6d4b20abeace85d P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:43.039384 12144 tablet_replica.cc:330] T adfc7405d029466ab16fdd8955554f76 P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:43.039546 12144 raft_consensus.cc:2229] T adfc7405d029466ab16fdd8955554f76 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:43.039667 12144 raft_consensus.cc:2258] T adfc7405d029466ab16fdd8955554f76 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:43.040272 12134 tablet_bootstrap.cc:492] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:43.040388 12134 ts_tablet_manager.cc:1367] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 01:59:43.040880 12134 raft_consensus.cc:348] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:43.041004 12134 raft_consensus.cc:374] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:43.041033 12134 raft_consensus.cc:729] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.041172 12134 consensus_queue.cc:260] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:43.041430 12134 ts_tablet_manager.cc:1398] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:43.041672 12134 tablet_bootstrap.cc:492] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:43.042140 12135 tablet_bootstrap.cc:492] T d53b8c06a6cb4ba9a6d4b20abeace85d P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:43.042245 12135 ts_tablet_manager.cc:1367] T d53b8c06a6cb4ba9a6d4b20abeace85d P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 01:59:43.042635 12134 tablet_bootstrap.cc:654] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:43.042765 12135 raft_consensus.cc:348] T d53b8c06a6cb4ba9a6d4b20abeace85d P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:43.042866 12135 raft_consensus.cc:374] T d53b8c06a6cb4ba9a6d4b20abeace85d P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:43.042925 12135 raft_consensus.cc:729] T d53b8c06a6cb4ba9a6d4b20abeace85d P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.043056 12135 consensus_queue.cc:260] T d53b8c06a6cb4ba9a6d4b20abeace85d P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:43.043375 12135 ts_tablet_manager.cc:1398] T d53b8c06a6cb4ba9a6d4b20abeace85d P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:43.043555 12135 tablet_bootstrap.cc:492] T aab7d8c559ff446192f471aacaa47584 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:43.044698 12135 tablet_bootstrap.cc:654] T aab7d8c559ff446192f471aacaa47584 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:43.045094 12134 tablet_bootstrap.cc:492] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:43.045185 12134 ts_tablet_manager.cc:1367] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 01:59:43.045728 12144 ts_tablet_manager.cc:1875] T adfc7405d029466ab16fdd8955554f76 P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:43.045717 12134 raft_consensus.cc:348] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:43.045816 12134 raft_consensus.cc:374] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:43.045873 12134 raft_consensus.cc:729] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.046104 12135 tablet_bootstrap.cc:492] T aab7d8c559ff446192f471aacaa47584 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:43.046062 12134 consensus_queue.cc:260] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:43.046218 12135 ts_tablet_manager.cc:1367] T aab7d8c559ff446192f471aacaa47584 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:43.046587 12134 ts_tablet_manager.cc:1398] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:43.047045 12136 tablet_bootstrap.cc:492] T eac1ed1eaeb04232b889f32ed1df1b40 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:43.046991 12135 raft_consensus.cc:348] T aab7d8c559ff446192f471aacaa47584 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:43.047142 12135 raft_consensus.cc:374] T aab7d8c559ff446192f471aacaa47584 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:43.047178 12135 raft_consensus.cc:729] T aab7d8c559ff446192f471aacaa47584 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.047343 12135 consensus_queue.cc:260] T aab7d8c559ff446192f471aacaa47584 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:43.047665 12135 ts_tablet_manager.cc:1398] T aab7d8c559ff446192f471aacaa47584 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:43.050930 12136 tablet_bootstrap.cc:654] T eac1ed1eaeb04232b889f32ed1df1b40 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:43.051602 12140 tablet_bootstrap.cc:492] T eac1ed1eaeb04232b889f32ed1df1b40 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:43.051836 12140 ts_tablet_manager.cc:1367] T eac1ed1eaeb04232b889f32ed1df1b40 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20240503 01:59:43.052146 12136 tablet_bootstrap.cc:492] T eac1ed1eaeb04232b889f32ed1df1b40 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:43.052273 12136 ts_tablet_manager.cc:1367] T eac1ed1eaeb04232b889f32ed1df1b40 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240503 01:59:43.052611 12144 ts_tablet_manager.cc:1888] T adfc7405d029466ab16fdd8955554f76 P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:43.052695 12144 log.cc:1196] T adfc7405d029466ab16fdd8955554f76 P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/adfc7405d029466ab16fdd8955554f76
I20240503 01:59:43.052822 12136 raft_consensus.cc:348] T eac1ed1eaeb04232b889f32ed1df1b40 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:43.052516 12140 raft_consensus.cc:348] T eac1ed1eaeb04232b889f32ed1df1b40 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:43.052968 12136 raft_consensus.cc:374] T eac1ed1eaeb04232b889f32ed1df1b40 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:43.053023 12144 ts_tablet_manager.cc:1909] T adfc7405d029466ab16fdd8955554f76 P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:43.053001 12140 raft_consensus.cc:374] T eac1ed1eaeb04232b889f32ed1df1b40 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:43.053032 12136 raft_consensus.cc:729] T eac1ed1eaeb04232b889f32ed1df1b40 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.053063 12140 raft_consensus.cc:729] T eac1ed1eaeb04232b889f32ed1df1b40 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.053226 12136 consensus_queue.cc:260] T eac1ed1eaeb04232b889f32ed1df1b40 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:43.053309 12145 tablet_replica.cc:330] T adfc7405d029466ab16fdd8955554f76 P db1a0000aaa346f79c0aa4b6c5105191: stopping tablet replica
I20240503 01:59:43.053393 12140 consensus_queue.cc:260] T eac1ed1eaeb04232b889f32ed1df1b40 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:43.053464 12145 raft_consensus.cc:2229] T adfc7405d029466ab16fdd8955554f76 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:43.053588 12145 raft_consensus.cc:2258] T adfc7405d029466ab16fdd8955554f76 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:43.053912 12140 ts_tablet_manager.cc:1398] T eac1ed1eaeb04232b889f32ed1df1b40 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:43.055064 12145 ts_tablet_manager.cc:1875] T adfc7405d029466ab16fdd8955554f76 P db1a0000aaa346f79c0aa4b6c5105191: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:43.055156 12137 tablet_bootstrap.cc:492] T aab7d8c559ff446192f471aacaa47584 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:43.055250 12137 ts_tablet_manager.cc:1367] T aab7d8c559ff446192f471aacaa47584 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.000s
I20240503 01:59:43.055744 12137 raft_consensus.cc:348] T aab7d8c559ff446192f471aacaa47584 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:43.055859 12137 raft_consensus.cc:374] T aab7d8c559ff446192f471aacaa47584 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:43.056118 12137 raft_consensus.cc:729] T aab7d8c559ff446192f471aacaa47584 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.056260 12137 consensus_queue.cc:260] T aab7d8c559ff446192f471aacaa47584 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:43.056561 12137 ts_tablet_manager.cc:1398] T aab7d8c559ff446192f471aacaa47584 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:43.056581 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet adfc7405d029466ab16fdd8955554f76 (table rebalancer_dynamic_table_new_name [id=76d0e56c39964d109a70c4b51a82da1d]) successfully deleted
I20240503 01:59:43.057364 12136 ts_tablet_manager.cc:1398] T eac1ed1eaeb04232b889f32ed1df1b40 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.005s user 0.000s sys 0.000s
I20240503 01:59:43.062237 12145 ts_tablet_manager.cc:1888] T adfc7405d029466ab16fdd8955554f76 P db1a0000aaa346f79c0aa4b6c5105191: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:43.062422 12145 log.cc:1196] T adfc7405d029466ab16fdd8955554f76 P db1a0000aaa346f79c0aa4b6c5105191: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/adfc7405d029466ab16fdd8955554f76
I20240503 01:59:43.062819 12145 ts_tablet_manager.cc:1909] T adfc7405d029466ab16fdd8955554f76 P db1a0000aaa346f79c0aa4b6c5105191: Deleting consensus metadata
I20240503 01:59:43.064031 6726 catalog_manager.cc:4830] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): tablet adfc7405d029466ab16fdd8955554f76 (table rebalancer_dynamic_table_new_name [id=76d0e56c39964d109a70c4b51a82da1d]) successfully deleted
I20240503 01:59:43.077237 12163 ts_tablet_manager.cc:902] T 72e090a6c18e4a13bbb343eb40a07694 P 12e8c683db054df3b75c6285b98a5ad8: Initiating tablet copy from peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:43.077854 12163 tablet_copy_client.cc:308] T 72e090a6c18e4a13bbb343eb40a07694 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:42943
I20240503 01:59:43.078330 6903 tablet_copy_service.cc:140] P 5dcc33cc621a4cbbae155f3230fb7c46: Received BeginTabletCopySession request for tablet 72e090a6c18e4a13bbb343eb40a07694 from peer 12e8c683db054df3b75c6285b98a5ad8 ({username='slave'} at 127.0.99.133:55147)
I20240503 01:59:43.078444 6903 tablet_copy_service.cc:161] P 5dcc33cc621a4cbbae155f3230fb7c46: Beginning new tablet copy session on tablet 72e090a6c18e4a13bbb343eb40a07694 from peer 12e8c683db054df3b75c6285b98a5ad8 at {username='slave'} at 127.0.99.133:55147: session id = 12e8c683db054df3b75c6285b98a5ad8-72e090a6c18e4a13bbb343eb40a07694
I20240503 01:59:43.079533 6903 tablet_copy_source_session.cc:205] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:43.080500 12163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e090a6c18e4a13bbb343eb40a07694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.083039 12163 tablet_copy_client.cc:783] T 72e090a6c18e4a13bbb343eb40a07694 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:43.083236 12163 tablet_copy_client.cc:647] T 72e090a6c18e4a13bbb343eb40a07694 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:43.087641 12163 tablet_copy_client.cc:523] T 72e090a6c18e4a13bbb343eb40a07694 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:43.089259 12163 tablet_bootstrap.cc:492] T 72e090a6c18e4a13bbb343eb40a07694 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:43.105902 11786 raft_consensus.cc:482] T d53b8c06a6cb4ba9a6d4b20abeace85d P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:43.106043 11786 raft_consensus.cc:504] T d53b8c06a6cb4ba9a6d4b20abeace85d P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:43.107012 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d53b8c06a6cb4ba9a6d4b20abeace85d" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:43.107165 9755 raft_consensus.cc:2454] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 01:59:43.106508 11786 leader_election.cc:288] T d53b8c06a6cb4ba9a6d4b20abeace85d P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:43.107439 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d53b8c06a6cb4ba9a6d4b20abeace85d" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:43.107466 9560 leader_election.cc:302] T d53b8c06a6cb4ba9a6d4b20abeace85d P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:43.107903 11786 raft_consensus.cc:2781] T d53b8c06a6cb4ba9a6d4b20abeace85d P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:43.107964 11786 raft_consensus.cc:482] T d53b8c06a6cb4ba9a6d4b20abeace85d P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:43.107568 9363 raft_consensus.cc:2454] T d53b8c06a6cb4ba9a6d4b20abeace85d P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 01:59:43.108013 11786 raft_consensus.cc:3037] T d53b8c06a6cb4ba9a6d4b20abeace85d P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:43.109102 11786 raft_consensus.cc:504] T d53b8c06a6cb4ba9a6d4b20abeace85d P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:43.109458 11786 leader_election.cc:288] T d53b8c06a6cb4ba9a6d4b20abeace85d P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Requested vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:43.110232 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d53b8c06a6cb4ba9a6d4b20abeace85d" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:43.110355 9755 raft_consensus.cc:3037] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:43.111506 9755 raft_consensus.cc:2454] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 01:59:43.112113 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d53b8c06a6cb4ba9a6d4b20abeace85d" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:43.112236 9363 raft_consensus.cc:3037] T d53b8c06a6cb4ba9a6d4b20abeace85d P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:43.113190 9363 raft_consensus.cc:2454] T d53b8c06a6cb4ba9a6d4b20abeace85d P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 01:59:43.113504 9560 leader_election.cc:302] T d53b8c06a6cb4ba9a6d4b20abeace85d P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:43.113986 11786 raft_consensus.cc:2781] T d53b8c06a6cb4ba9a6d4b20abeace85d P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:43.114084 11786 raft_consensus.cc:686] T d53b8c06a6cb4ba9a6d4b20abeace85d P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 LEADER]: Becoming Leader. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Running, Role: LEADER
I20240503 01:59:43.114233 11786 consensus_queue.cc:237] T d53b8c06a6cb4ba9a6d4b20abeace85d P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:43.116379 6735 catalog_manager.cc:5484] T d53b8c06a6cb4ba9a6d4b20abeace85d P b7062fbbc2b54fff9e99ec4ea5562c76 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:43.124794 12163 tablet_bootstrap.cc:492] T 72e090a6c18e4a13bbb343eb40a07694 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:43.125221 12163 tablet_bootstrap.cc:492] T 72e090a6c18e4a13bbb343eb40a07694 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap complete.
I20240503 01:59:43.125504 12163 ts_tablet_manager.cc:1367] T 72e090a6c18e4a13bbb343eb40a07694 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.036s user 0.016s sys 0.014s
I20240503 01:59:43.126220 12163 raft_consensus.cc:348] T 72e090a6c18e4a13bbb343eb40a07694 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:43.126475 12163 raft_consensus.cc:729] T 72e090a6c18e4a13bbb343eb40a07694 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.126726 12163 consensus_queue.cc:260] T 72e090a6c18e4a13bbb343eb40a07694 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:43.127828 12163 ts_tablet_manager.cc:1398] T 72e090a6c18e4a13bbb343eb40a07694 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:43.128386 6903 tablet_copy_service.cc:342] P 5dcc33cc621a4cbbae155f3230fb7c46: Request end of tablet copy session 12e8c683db054df3b75c6285b98a5ad8-72e090a6c18e4a13bbb343eb40a07694 received from {username='slave'} at 127.0.99.133:55147
I20240503 01:59:43.128471 6903 tablet_copy_service.cc:433] P 5dcc33cc621a4cbbae155f3230fb7c46: ending tablet copy session 12e8c683db054df3b75c6285b98a5ad8-72e090a6c18e4a13bbb343eb40a07694 on tablet 72e090a6c18e4a13bbb343eb40a07694 with peer 12e8c683db054df3b75c6285b98a5ad8
I20240503 01:59:43.131167 6733 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:43.140480 9473 tablet_service.cc:1467] Processing CreateTablet for tablet f68d6cd3a9bd4b67b5fe23af57878b81 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f4d9011826fe438984b268a77313e24e]), partition=
I20240503 01:59:43.140492 9737 tablet_service.cc:1467] Processing CreateTablet for tablet f68d6cd3a9bd4b67b5fe23af57878b81 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f4d9011826fe438984b268a77313e24e]), partition=
I20240503 01:59:43.140766 9737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68d6cd3a9bd4b67b5fe23af57878b81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.140766 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68d6cd3a9bd4b67b5fe23af57878b81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.142962 12134 tablet_bootstrap.cc:492] T f68d6cd3a9bd4b67b5fe23af57878b81 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:43.143895 12134 tablet_bootstrap.cc:654] T f68d6cd3a9bd4b67b5fe23af57878b81 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:43.144850 9604 tablet_service.cc:1467] Processing CreateTablet for tablet f68d6cd3a9bd4b67b5fe23af57878b81 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f4d9011826fe438984b268a77313e24e]), partition=
I20240503 01:59:43.145110 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f68d6cd3a9bd4b67b5fe23af57878b81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.147398 12136 tablet_bootstrap.cc:492] T f68d6cd3a9bd4b67b5fe23af57878b81 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:43.147409 12140 tablet_bootstrap.cc:492] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:43.148408 12136 tablet_bootstrap.cc:654] T f68d6cd3a9bd4b67b5fe23af57878b81 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:43.148476 12140 tablet_bootstrap.cc:654] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:43.149880 12136 tablet_bootstrap.cc:492] T f68d6cd3a9bd4b67b5fe23af57878b81 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:43.149972 12136 ts_tablet_manager.cc:1367] T f68d6cd3a9bd4b67b5fe23af57878b81 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 01:59:43.150123 12140 tablet_bootstrap.cc:492] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:43.150210 12140 ts_tablet_manager.cc:1367] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:43.150792 12140 raft_consensus.cc:348] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:43.150913 12140 raft_consensus.cc:374] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:43.150955 12140 raft_consensus.cc:729] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.150914 12136 raft_consensus.cc:348] T f68d6cd3a9bd4b67b5fe23af57878b81 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:43.151026 12136 raft_consensus.cc:374] T f68d6cd3a9bd4b67b5fe23af57878b81 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:43.151050 12136 raft_consensus.cc:729] T f68d6cd3a9bd4b67b5fe23af57878b81 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.151113 12140 consensus_queue.cc:260] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:43.151170 12136 consensus_queue.cc:260] T f68d6cd3a9bd4b67b5fe23af57878b81 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:43.151464 12140 ts_tablet_manager.cc:1398] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:43.151469 12136 ts_tablet_manager.cc:1398] T f68d6cd3a9bd4b67b5fe23af57878b81 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:43.153944 12134 tablet_bootstrap.cc:492] T f68d6cd3a9bd4b67b5fe23af57878b81 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:43.154062 12134 ts_tablet_manager.cc:1367] T f68d6cd3a9bd4b67b5fe23af57878b81 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 01:59:43.154626 12203 ts_tablet_manager.cc:902] T d2fbadb911aa4bd7b4db021860c202d3 P db1a0000aaa346f79c0aa4b6c5105191: Initiating tablet copy from peer 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:43.154585 12134 raft_consensus.cc:348] T f68d6cd3a9bd4b67b5fe23af57878b81 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:43.154706 12134 raft_consensus.cc:374] T f68d6cd3a9bd4b67b5fe23af57878b81 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:43.154738 12134 raft_consensus.cc:729] T f68d6cd3a9bd4b67b5fe23af57878b81 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.154862 12134 consensus_queue.cc:260] T f68d6cd3a9bd4b67b5fe23af57878b81 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:43.155180 12134 ts_tablet_manager.cc:1398] T f68d6cd3a9bd4b67b5fe23af57878b81 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:43.155158 12203 tablet_copy_client.cc:308] T d2fbadb911aa4bd7b4db021860c202d3 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36271
I20240503 01:59:43.162374 7295 tablet_copy_service.cc:140] P 1d0e738c34af40fa9950c6f086619b3a: Received BeginTabletCopySession request for tablet d2fbadb911aa4bd7b4db021860c202d3 from peer db1a0000aaa346f79c0aa4b6c5105191 ({username='slave'} at 127.0.99.136:38163)
I20240503 01:59:43.162546 7295 tablet_copy_service.cc:161] P 1d0e738c34af40fa9950c6f086619b3a: Beginning new tablet copy session on tablet d2fbadb911aa4bd7b4db021860c202d3 from peer db1a0000aaa346f79c0aa4b6c5105191 at {username='slave'} at 127.0.99.136:38163: session id = db1a0000aaa346f79c0aa4b6c5105191-d2fbadb911aa4bd7b4db021860c202d3
I20240503 01:59:43.163578 7295 tablet_copy_source_session.cc:205] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:43.164616 12203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2fbadb911aa4bd7b4db021860c202d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.167050 12203 tablet_copy_client.cc:783] T d2fbadb911aa4bd7b4db021860c202d3 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:43.167218 12203 tablet_copy_client.cc:647] T d2fbadb911aa4bd7b4db021860c202d3 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:43.170264 12203 tablet_copy_client.cc:523] T d2fbadb911aa4bd7b4db021860c202d3 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:43.172814 12203 tablet_bootstrap.cc:492] T d2fbadb911aa4bd7b4db021860c202d3 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:43.191109 12211 ts_tablet_manager.cc:902] T 2f271b67a6f34eb18c98824ca4b26d17 P 1d0e738c34af40fa9950c6f086619b3a: Initiating tablet copy from peer db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:43.192093 12211 tablet_copy_client.cc:308] T 2f271b67a6f34eb18c98824ca4b26d17 P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:42053
I20240503 01:59:43.192729 9777 tablet_copy_service.cc:140] P db1a0000aaa346f79c0aa4b6c5105191: Received BeginTabletCopySession request for tablet 2f271b67a6f34eb18c98824ca4b26d17 from peer 1d0e738c34af40fa9950c6f086619b3a ({username='slave'} at 127.0.99.132:41445)
I20240503 01:59:43.192845 9777 tablet_copy_service.cc:161] P db1a0000aaa346f79c0aa4b6c5105191: Beginning new tablet copy session on tablet 2f271b67a6f34eb18c98824ca4b26d17 from peer 1d0e738c34af40fa9950c6f086619b3a at {username='slave'} at 127.0.99.132:41445: session id = 1d0e738c34af40fa9950c6f086619b3a-2f271b67a6f34eb18c98824ca4b26d17
I20240503 01:59:43.193790 9777 tablet_copy_source_session.cc:205] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:43.195340 12211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f271b67a6f34eb18c98824ca4b26d17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.197968 12211 tablet_copy_client.cc:783] T 2f271b67a6f34eb18c98824ca4b26d17 P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:43.198197 12211 tablet_copy_client.cc:647] T 2f271b67a6f34eb18c98824ca4b26d17 P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:43.200078 12216 ts_tablet_manager.cc:902] T 76e8697ac8624e03a46bddae35f4c9fa P bb5364d730cb4489a0bfc6c270b7450a: Initiating tablet copy from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:43.200256 12211 tablet_copy_client.cc:523] T 2f271b67a6f34eb18c98824ca4b26d17 P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:43.200644 12216 tablet_copy_client.cc:308] T 76e8697ac8624e03a46bddae35f4c9fa P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41979
I20240503 01:59:43.201176 7165 tablet_copy_service.cc:140] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Received BeginTabletCopySession request for tablet 76e8697ac8624e03a46bddae35f4c9fa from peer bb5364d730cb4489a0bfc6c270b7450a ({username='slave'} at 127.0.99.134:59853)
I20240503 01:59:43.201295 7165 tablet_copy_service.cc:161] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Beginning new tablet copy session on tablet 76e8697ac8624e03a46bddae35f4c9fa from peer bb5364d730cb4489a0bfc6c270b7450a at {username='slave'} at 127.0.99.134:59853: session id = bb5364d730cb4489a0bfc6c270b7450a-76e8697ac8624e03a46bddae35f4c9fa
I20240503 01:59:43.202071 12203 tablet_bootstrap.cc:492] T d2fbadb911aa4bd7b4db021860c202d3 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:43.202235 12211 tablet_bootstrap.cc:492] T 2f271b67a6f34eb18c98824ca4b26d17 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:43.202289 7165 tablet_copy_source_session.cc:205] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:43.202378 12203 tablet_bootstrap.cc:492] T d2fbadb911aa4bd7b4db021860c202d3 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap complete.
I20240503 01:59:43.202601 12203 ts_tablet_manager.cc:1367] T d2fbadb911aa4bd7b4db021860c202d3 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.030s user 0.027s sys 0.002s
I20240503 01:59:43.202891 11989 raft_consensus.cc:482] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:43.202973 11989 raft_consensus.cc:504] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:43.203123 12203 raft_consensus.cc:348] T d2fbadb911aa4bd7b4db021860c202d3 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:43.203213 12203 raft_consensus.cc:729] T d2fbadb911aa4bd7b4db021860c202d3 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.203327 12203 consensus_queue.cc:260] T d2fbadb911aa4bd7b4db021860c202d3 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:43.203377 11989 leader_election.cc:288] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:43.203768 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aab7d8c559ff446192f471aacaa47584" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:43.203909 9363 raft_consensus.cc:2454] T aab7d8c559ff446192f471aacaa47584 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:43.204185 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aab7d8c559ff446192f471aacaa47584" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:43.204260 7211 leader_election.cc:302] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 1d0e738c34af40fa9950c6f086619b3a; no voters:
I20240503 01:59:43.204336 6882 raft_consensus.cc:2454] T aab7d8c559ff446192f471aacaa47584 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:43.204545 12203 ts_tablet_manager.cc:1398] T d2fbadb911aa4bd7b4db021860c202d3 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:43.205071 7295 tablet_copy_service.cc:342] P 1d0e738c34af40fa9950c6f086619b3a: Request end of tablet copy session db1a0000aaa346f79c0aa4b6c5105191-d2fbadb911aa4bd7b4db021860c202d3 received from {username='slave'} at 127.0.99.136:38163
I20240503 01:59:43.205173 7295 tablet_copy_service.cc:433] P 1d0e738c34af40fa9950c6f086619b3a: ending tablet copy session db1a0000aaa346f79c0aa4b6c5105191-d2fbadb911aa4bd7b4db021860c202d3 on tablet d2fbadb911aa4bd7b4db021860c202d3 with peer db1a0000aaa346f79c0aa4b6c5105191
I20240503 01:59:43.205411 12216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76e8697ac8624e03a46bddae35f4c9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.205535 11989 raft_consensus.cc:2781] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:43.205595 11989 raft_consensus.cc:482] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:43.205646 11989 raft_consensus.cc:3037] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:43.206755 11989 raft_consensus.cc:504] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:43.207165 11989 leader_election.cc:288] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Requested vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:43.207631 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aab7d8c559ff446192f471aacaa47584" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:43.207769 6882 raft_consensus.cc:3037] T aab7d8c559ff446192f471aacaa47584 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:43.208004 12216 tablet_copy_client.cc:783] T 76e8697ac8624e03a46bddae35f4c9fa P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:43.208159 12216 tablet_copy_client.cc:647] T 76e8697ac8624e03a46bddae35f4c9fa P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:43.208932 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aab7d8c559ff446192f471aacaa47584" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:43.208953 6882 raft_consensus.cc:2454] T aab7d8c559ff446192f471aacaa47584 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:43.209038 9363 raft_consensus.cc:3037] T aab7d8c559ff446192f471aacaa47584 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:43.209306 7212 leader_election.cc:302] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:43.209599 11989 raft_consensus.cc:2781] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:43.209825 11989 raft_consensus.cc:686] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 01:59:43.209988 11989 consensus_queue.cc:237] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:43.210073 9363 raft_consensus.cc:2454] T aab7d8c559ff446192f471aacaa47584 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:43.210142 12216 tablet_copy_client.cc:523] T 76e8697ac8624e03a46bddae35f4c9fa P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:43.212380 12216 tablet_bootstrap.cc:492] T 76e8697ac8624e03a46bddae35f4c9fa P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:43.213018 6737 catalog_manager.cc:5484] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:43.216929 12211 tablet_bootstrap.cc:492] T 2f271b67a6f34eb18c98824ca4b26d17 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:43.217218 12211 tablet_bootstrap.cc:492] T 2f271b67a6f34eb18c98824ca4b26d17 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap complete.
I20240503 01:59:43.217383 12211 ts_tablet_manager.cc:1367] T 2f271b67a6f34eb18c98824ca4b26d17 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240503 01:59:43.218119 12211 raft_consensus.cc:348] T 2f271b67a6f34eb18c98824ca4b26d17 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:43.218247 12211 raft_consensus.cc:729] T 2f271b67a6f34eb18c98824ca4b26d17 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.218376 12211 consensus_queue.cc:260] T 2f271b67a6f34eb18c98824ca4b26d17 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:43.218815 12211 ts_tablet_manager.cc:1398] T 2f271b67a6f34eb18c98824ca4b26d17 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:43.219334 9777 tablet_copy_service.cc:342] P db1a0000aaa346f79c0aa4b6c5105191: Request end of tablet copy session 1d0e738c34af40fa9950c6f086619b3a-2f271b67a6f34eb18c98824ca4b26d17 received from {username='slave'} at 127.0.99.132:41445
I20240503 01:59:43.219422 9777 tablet_copy_service.cc:433] P db1a0000aaa346f79c0aa4b6c5105191: ending tablet copy session 1d0e738c34af40fa9950c6f086619b3a-2f271b67a6f34eb18c98824ca4b26d17 on tablet 2f271b67a6f34eb18c98824ca4b26d17 with peer 1d0e738c34af40fa9950c6f086619b3a
I20240503 01:59:43.225088 11326 raft_consensus.cc:482] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:43.225234 11326 raft_consensus.cc:504] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:43.225661 11326 leader_election.cc:288] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:43.226212 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eac1ed1eaeb04232b889f32ed1df1b40" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:43.226363 9494 raft_consensus.cc:2454] T eac1ed1eaeb04232b889f32ed1df1b40 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 01:59:43.226389 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eac1ed1eaeb04232b889f32ed1df1b40" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:43.226580 9625 raft_consensus.cc:2454] T eac1ed1eaeb04232b889f32ed1df1b40 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 01:59:43.226939 9692 leader_election.cc:302] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:43.227242 11326 raft_consensus.cc:2781] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:43.227305 11326 raft_consensus.cc:482] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:43.227345 11326 raft_consensus.cc:3037] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:43.228380 11326 raft_consensus.cc:504] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:43.228760 11326 leader_election.cc:288] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Requested vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:43.229385 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eac1ed1eaeb04232b889f32ed1df1b40" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:43.229387 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eac1ed1eaeb04232b889f32ed1df1b40" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:43.229600 9494 raft_consensus.cc:3037] T eac1ed1eaeb04232b889f32ed1df1b40 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:43.229600 9625 raft_consensus.cc:3037] T eac1ed1eaeb04232b889f32ed1df1b40 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:43.230846 9494 raft_consensus.cc:2454] T eac1ed1eaeb04232b889f32ed1df1b40 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:43.230896 9625 raft_consensus.cc:2454] T eac1ed1eaeb04232b889f32ed1df1b40 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:43.231230 9690 leader_election.cc:302] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:43.231411 11326 raft_consensus.cc:2781] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:43.231537 11326 raft_consensus.cc:686] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Becoming Leader. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Running, Role: LEADER
I20240503 01:59:43.231681 11326 consensus_queue.cc:237] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:43.234218 6737 catalog_manager.cc:5484] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: term changed from 0 to 1, leader changed from <none> to db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:43.238011 12216 tablet_bootstrap.cc:492] T 76e8697ac8624e03a46bddae35f4c9fa P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:43.238324 12216 tablet_bootstrap.cc:492] T 76e8697ac8624e03a46bddae35f4c9fa P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap complete.
I20240503 01:59:43.238505 12216 ts_tablet_manager.cc:1367] T 76e8697ac8624e03a46bddae35f4c9fa P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.026s user 0.015s sys 0.008s
I20240503 01:59:43.238981 12216 raft_consensus.cc:348] T 76e8697ac8624e03a46bddae35f4c9fa P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:43.239120 12216 raft_consensus.cc:729] T 76e8697ac8624e03a46bddae35f4c9fa P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.239243 12216 consensus_queue.cc:260] T 76e8697ac8624e03a46bddae35f4c9fa P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:43.239526 12216 ts_tablet_manager.cc:1398] T 76e8697ac8624e03a46bddae35f4c9fa P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:43.240013 7165 tablet_copy_service.cc:342] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Request end of tablet copy session bb5364d730cb4489a0bfc6c270b7450a-76e8697ac8624e03a46bddae35f4c9fa received from {username='slave'} at 127.0.99.134:59853
I20240503 01:59:43.240110 7165 tablet_copy_service.cc:433] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: ending tablet copy session bb5364d730cb4489a0bfc6c270b7450a-76e8697ac8624e03a46bddae35f4c9fa on tablet 76e8697ac8624e03a46bddae35f4c9fa with peer bb5364d730cb4489a0bfc6c270b7450a
I20240503 01:59:43.248733 12070 raft_consensus.cc:482] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:43.248870 12070 raft_consensus.cc:504] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:43.249281 12070 leader_election.cc:288] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:43.250250 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68d6cd3a9bd4b67b5fe23af57878b81" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:43.250403 9625 raft_consensus.cc:2454] T f68d6cd3a9bd4b67b5fe23af57878b81 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:43.251194 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68d6cd3a9bd4b67b5fe23af57878b81" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:43.251333 9755 raft_consensus.cc:2454] T f68d6cd3a9bd4b67b5fe23af57878b81 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:43.251657 9428 leader_election.cc:302] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:43.253466 12070 raft_consensus.cc:2781] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:43.253537 12070 raft_consensus.cc:482] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:43.253590 12070 raft_consensus.cc:3037] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:43.254683 6863 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:43.259702 12070 raft_consensus.cc:504] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:43.260138 12070 leader_election.cc:288] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:43.260746 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68d6cd3a9bd4b67b5fe23af57878b81" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:43.260862 9755 raft_consensus.cc:3037] T f68d6cd3a9bd4b67b5fe23af57878b81 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:43.261051 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f68d6cd3a9bd4b67b5fe23af57878b81" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:43.261188 9625 raft_consensus.cc:3037] T f68d6cd3a9bd4b67b5fe23af57878b81 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:43.261710 9755 raft_consensus.cc:2454] T f68d6cd3a9bd4b67b5fe23af57878b81 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:43.262156 9625 raft_consensus.cc:2454] T f68d6cd3a9bd4b67b5fe23af57878b81 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:43.262575 9428 leader_election.cc:302] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:43.262945 12070 raft_consensus.cc:2781] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:43.263025 12070 raft_consensus.cc:686] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Becoming Leader. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Running, Role: LEADER
I20240503 01:59:43.263168 12070 consensus_queue.cc:237] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:43.265753 6733 catalog_manager.cc:5484] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: term changed from 0 to 1, leader changed from <none> to bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:43.266482 7256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:43.269404 9342 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:43.270251 6993 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:43.270709 9604 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:43.273165 12211 ts_tablet_manager.cc:902] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a: Initiating tablet copy from peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:43.273950 12211 tablet_copy_client.cc:308] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:46087
I20240503 01:59:43.274823 9646 tablet_copy_service.cc:140] P b7062fbbc2b54fff9e99ec4ea5562c76: Received BeginTabletCopySession request for tablet 1d182ce2c5084082a0990ce8090eb88b from peer 1d0e738c34af40fa9950c6f086619b3a ({username='slave'} at 127.0.99.132:52863)
I20240503 01:59:43.274961 9646 tablet_copy_service.cc:161] P b7062fbbc2b54fff9e99ec4ea5562c76: Beginning new tablet copy session on tablet 1d182ce2c5084082a0990ce8090eb88b from peer 1d0e738c34af40fa9950c6f086619b3a at {username='slave'} at 127.0.99.132:52863: session id = 1d0e738c34af40fa9950c6f086619b3a-1d182ce2c5084082a0990ce8090eb88b
I20240503 01:59:43.276232 9473 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:43.276293 9646 tablet_copy_source_session.cc:205] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:43.276810 7124 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:43.283978 12211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d182ce2c5084082a0990ce8090eb88b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.286325 12211 tablet_copy_client.cc:783] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:43.286484 12211 tablet_copy_client.cc:647] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:43.292187 12211 tablet_copy_client.cc:523] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:43.293834 12211 tablet_bootstrap.cc:492] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:43.307865 9737 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:43.320008 12211 tablet_bootstrap.cc:492] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:43.320297 12211 tablet_bootstrap.cc:492] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap complete.
I20240503 01:59:43.320456 12211 ts_tablet_manager.cc:1367] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.000s
I20240503 01:59:43.320960 12211 raft_consensus.cc:348] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:43.321081 12211 raft_consensus.cc:729] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.321208 12211 consensus_queue.cc:260] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:43.321481 12211 ts_tablet_manager.cc:1398] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:43.322350 9646 tablet_copy_service.cc:342] P b7062fbbc2b54fff9e99ec4ea5562c76: Request end of tablet copy session 1d0e738c34af40fa9950c6f086619b3a-1d182ce2c5084082a0990ce8090eb88b received from {username='slave'} at 127.0.99.132:52863
I20240503 01:59:43.322463 9646 tablet_copy_service.cc:433] P b7062fbbc2b54fff9e99ec4ea5562c76: ending tablet copy session 1d0e738c34af40fa9950c6f086619b3a-1d182ce2c5084082a0990ce8090eb88b on tablet 1d182ce2c5084082a0990ce8090eb88b with peer 1d0e738c34af40fa9950c6f086619b3a
I20240503 01:59:43.339032 12243 ts_tablet_manager.cc:902] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef: Initiating tablet copy from peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:43.340150 12243 tablet_copy_client.cc:308] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:43471
I20240503 01:59:43.340909 9514 tablet_copy_service.cc:140] P bb5364d730cb4489a0bfc6c270b7450a: Received BeginTabletCopySession request for tablet 31f5289f1dff4fcaa3492c16e40a3ed7 from peer 63331f455be049c1bdd6c23c59ac1bef ({username='slave'} at 127.0.99.130:54311)
I20240503 01:59:43.341040 9514 tablet_copy_service.cc:161] P bb5364d730cb4489a0bfc6c270b7450a: Beginning new tablet copy session on tablet 31f5289f1dff4fcaa3492c16e40a3ed7 from peer 63331f455be049c1bdd6c23c59ac1bef at {username='slave'} at 127.0.99.130:54311: session id = 63331f455be049c1bdd6c23c59ac1bef-31f5289f1dff4fcaa3492c16e40a3ed7
I20240503 01:59:43.344043 9514 tablet_copy_source_session.cc:205] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:43.346974 12243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f5289f1dff4fcaa3492c16e40a3ed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.349941 12243 tablet_copy_client.cc:783] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:43.350224 12243 tablet_copy_client.cc:647] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:43.352027 12243 tablet_copy_client.cc:523] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:43.353524 12243 tablet_bootstrap.cc:492] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:43.357972 12243 tablet_bootstrap.cc:492] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:43.358779 12243 tablet_bootstrap.cc:492] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap complete.
I20240503 01:59:43.359031 12243 ts_tablet_manager.cc:1367] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.006s
I20240503 01:59:43.359547 12243 raft_consensus.cc:348] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:43.359730 12243 raft_consensus.cc:729] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.359928 12243 consensus_queue.cc:260] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:43.360335 12243 ts_tablet_manager.cc:1398] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:43.361152 9514 tablet_copy_service.cc:342] P bb5364d730cb4489a0bfc6c270b7450a: Request end of tablet copy session 63331f455be049c1bdd6c23c59ac1bef-31f5289f1dff4fcaa3492c16e40a3ed7 received from {username='slave'} at 127.0.99.130:54311
I20240503 01:59:43.361244 9514 tablet_copy_service.cc:433] P bb5364d730cb4489a0bfc6c270b7450a: ending tablet copy session 63331f455be049c1bdd6c23c59ac1bef-31f5289f1dff4fcaa3492c16e40a3ed7 on tablet 31f5289f1dff4fcaa3492c16e40a3ed7 with peer 63331f455be049c1bdd6c23c59ac1bef
I20240503 01:59:43.401566 6733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:43.406963 9755 raft_consensus.cc:1261] T f68d6cd3a9bd4b67b5fe23af57878b81 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:43.407539 9626 raft_consensus.cc:1261] T f68d6cd3a9bd4b67b5fe23af57878b81 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:43.407584 12070 consensus_queue.cc:1035] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:43.407943 12070 consensus_queue.cc:1035] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:43.411319 6737 catalog_manager.cc:5576] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053) does not have the latest schema for tablet f68d6cd3a9bd4b67b5fe23af57878b81 (table rebalancer_dynamic_table [id=f4d9011826fe438984b268a77313e24e]). Expected version 1 got 0
I20240503 01:59:43.411885 11986 tablet.cc:1722] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:43.412231 11986 tablet.cc:1620] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:43.412182 12292 tablet.cc:1722] T f68d6cd3a9bd4b67b5fe23af57878b81 P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:43.412384 6733 catalog_manager.cc:5576] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087) does not have the latest schema for tablet f68d6cd3a9bd4b67b5fe23af57878b81 (table rebalancer_dynamic_table [id=f4d9011826fe438984b268a77313e24e]). Expected version 1 got 0
I20240503 01:59:43.412564 12292 tablet.cc:1620] T f68d6cd3a9bd4b67b5fe23af57878b81 P b7062fbbc2b54fff9e99ec4ea5562c76: MemRowSet was empty: no flush needed.
I20240503 01:59:43.417363 6737 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=f4d9011826fe438984b268a77313e24e] alter complete (version 1)
I20240503 01:59:43.422610 12294 tablet.cc:1722] T f68d6cd3a9bd4b67b5fe23af57878b81 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:43.423135 12294 tablet.cc:1620] T f68d6cd3a9bd4b67b5fe23af57878b81 P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:43.424641 6733 catalog_manager.cc:5576] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053) does not have the latest schema for tablet f68d6cd3a9bd4b67b5fe23af57878b81 (table rebalancer_dynamic_table [id=f4d9011826fe438984b268a77313e24e]). Expected version 1 got 0
I20240503 01:59:43.463305 9342 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:43.467797 9604 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:43.468340 9363 raft_consensus.cc:1203] T 72e090a6c18e4a13bbb343eb40a07694 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:43.470624 9473 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:43.471719 6863 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:43.472206 7124 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:43.472633 6993 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:43.473112 7256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:43.506935 9737 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:43.519498 6732 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:43.524155 11786 consensus_queue.cc:1035] T d53b8c06a6cb4ba9a6d4b20abeace85d P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:43.526997 11986 tablet.cc:1722] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:43.527385 11986 tablet.cc:1620] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:43.529589 12292 tablet.cc:1722] T f68d6cd3a9bd4b67b5fe23af57878b81 P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:43.529970 12292 tablet.cc:1620] T f68d6cd3a9bd4b67b5fe23af57878b81 P b7062fbbc2b54fff9e99ec4ea5562c76: MemRowSet was empty: no flush needed.
I20240503 01:59:43.530429 12294 tablet.cc:1722] T f68d6cd3a9bd4b67b5fe23af57878b81 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:43.530850 12294 tablet.cc:1620] T f68d6cd3a9bd4b67b5fe23af57878b81 P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:43.534655 6740 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=f4d9011826fe438984b268a77313e24e] alter complete (version 2)
I20240503 01:59:43.539341 11786 consensus_queue.cc:1035] T d53b8c06a6cb4ba9a6d4b20abeace85d P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:43.543466 9755 raft_consensus.cc:1203] T d2fbadb911aa4bd7b4db021860c202d3 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:43.639523 6740 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:43.648268 11986 tablet.cc:1722] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:43.648948 9604 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:43.650439 9494 raft_consensus.cc:1203] T 76e8697ac8624e03a46bddae35f4c9fa P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:43.653800 12292 tablet.cc:1722] T f68d6cd3a9bd4b67b5fe23af57878b81 P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:43.655316 11326 consensus_queue.cc:1035] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:43.656240 12294 tablet.cc:1722] T f68d6cd3a9bd4b67b5fe23af57878b81 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:43.660787 9342 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:43.663496 6993 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:43.668756 6740 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=f4d9011826fe438984b268a77313e24e] alter complete (version 3)
I20240503 01:59:43.675426 12011 consensus_queue.cc:1035] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:43.677398 9737 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:43.679447 7256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:43.681149 7124 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:43.682037 9473 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:43.690845 6863 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:43.705279 7274 raft_consensus.cc:1203] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 01:59:43.708077 7274 raft_consensus.cc:1203] T 2f271b67a6f34eb18c98824ca4b26d17 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 01:59:43.727270 12221 consensus_queue.cc:1035] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:43.730577 12366 consensus_queue.cc:1035] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:43.742902 7014 raft_consensus.cc:1203] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 01:59:43.758229 7274 consensus_queue.cc:237] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:43.759799 9755 raft_consensus.cc:1261] T d2fbadb911aa4bd7b4db021860c202d3 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:43.760223 7143 raft_consensus.cc:1261] T d2fbadb911aa4bd7b4db021860c202d3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:43.760279 12366 consensus_queue.cc:1035] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:43.760921 12366 consensus_queue.cc:1035] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:43.762173 12221 raft_consensus.cc:2932] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:43.762614 9755 raft_consensus.cc:2932] T d2fbadb911aa4bd7b4db021860c202d3 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:43.766402 6740 catalog_manager.cc:5484] T d2fbadb911aa4bd7b4db021860c202d3 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: config changed from index 4 to 5, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:43.769381 7143 raft_consensus.cc:2932] T d2fbadb911aa4bd7b4db021860c202d3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:43.770346 6993 tablet_service.cc:1514] Processing DeleteTablet for tablet d2fbadb911aa4bd7b4db021860c202d3 with delete_type TABLET_DATA_TOMBSTONED (TS 63331f455be049c1bdd6c23c59ac1bef not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:43.772331 6732 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:43.775770 6740 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:43.775875 12394 tablet_replica.cc:330] T d2fbadb911aa4bd7b4db021860c202d3 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:43.775911 6740 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:43.776032 12394 raft_consensus.cc:2229] T d2fbadb911aa4bd7b4db021860c202d3 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:43.776199 12394 raft_consensus.cc:2258] T d2fbadb911aa4bd7b4db021860c202d3 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:43.776710 12394 ts_tablet_manager.cc:1875] T d2fbadb911aa4bd7b4db021860c202d3 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:43.781502 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet f68d6cd3a9bd4b67b5fe23af57878b81
I20240503 01:59:43.782080 9473 tablet_service.cc:1514] Processing DeleteTablet for tablet f68d6cd3a9bd4b67b5fe23af57878b81 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:43 UTC) from {username='slave'} at 127.0.0.1:54678
I20240503 01:59:43.782335 9604 tablet_service.cc:1514] Processing DeleteTablet for tablet f68d6cd3a9bd4b67b5fe23af57878b81 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:43 UTC) from {username='slave'} at 127.0.0.1:58562
I20240503 01:59:43.782585 9737 tablet_service.cc:1514] Processing DeleteTablet for tablet f68d6cd3a9bd4b67b5fe23af57878b81 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:43 UTC) from {username='slave'} at 127.0.0.1:48152
I20240503 01:59:43.782630 12402 tablet_replica.cc:330] T f68d6cd3a9bd4b67b5fe23af57878b81 P b7062fbbc2b54fff9e99ec4ea5562c76: stopping tablet replica
I20240503 01:59:43.782778 12402 raft_consensus.cc:2229] T f68d6cd3a9bd4b67b5fe23af57878b81 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:43.782914 12394 ts_tablet_manager.cc:1888] T d2fbadb911aa4bd7b4db021860c202d3 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 01:59:43.782941 12402 raft_consensus.cc:2258] T f68d6cd3a9bd4b67b5fe23af57878b81 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:43.783057 12394 log.cc:1196] T d2fbadb911aa4bd7b4db021860c202d3 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/d2fbadb911aa4bd7b4db021860c202d3
I20240503 01:59:43.783203 12403 tablet_replica.cc:330] T f68d6cd3a9bd4b67b5fe23af57878b81 P db1a0000aaa346f79c0aa4b6c5105191: stopping tablet replica
I20240503 01:59:43.783635 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 9e21d082e40540d4bb908fc9e90055c9 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=9c9f7f12e036416d8388508bfb61c799]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:43.783751 9473 tablet_service.cc:1467] Processing CreateTablet for tablet a1879ffdfd2e4b26b894542d2f13a7c7 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=9c9f7f12e036416d8388508bfb61c799]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:43.783941 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e21d082e40540d4bb908fc9e90055c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.783994 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1879ffdfd2e4b26b894542d2f13a7c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.785408 6863 tablet_service.cc:1467] Processing CreateTablet for tablet 9e21d082e40540d4bb908fc9e90055c9 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=9c9f7f12e036416d8388508bfb61c799]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:43.785661 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e21d082e40540d4bb908fc9e90055c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.786641 12402 ts_tablet_manager.cc:1875] T f68d6cd3a9bd4b67b5fe23af57878b81 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:43.786702 9604 tablet_service.cc:1467] Processing CreateTablet for tablet a1879ffdfd2e4b26b894542d2f13a7c7 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=9c9f7f12e036416d8388508bfb61c799]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:43.787216 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1879ffdfd2e4b26b894542d2f13a7c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.789292 12402 ts_tablet_manager.cc:1888] T f68d6cd3a9bd4b67b5fe23af57878b81 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:43.789376 12402 log.cc:1196] T f68d6cd3a9bd4b67b5fe23af57878b81 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/f68d6cd3a9bd4b67b5fe23af57878b81
I20240503 01:59:43.789368 6861 tablet_service.cc:1467] Processing CreateTablet for tablet a1879ffdfd2e4b26b894542d2f13a7c7 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=9c9f7f12e036416d8388508bfb61c799]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:43.789685 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1879ffdfd2e4b26b894542d2f13a7c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.789705 12402 ts_tablet_manager.cc:1909] T f68d6cd3a9bd4b67b5fe23af57878b81 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting consensus metadata
I20240503 01:59:43.790016 12405 tablet_bootstrap.cc:492] T 9e21d082e40540d4bb908fc9e90055c9 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:43.792117 6727 catalog_manager.cc:4830] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): tablet f68d6cd3a9bd4b67b5fe23af57878b81 (table rebalancer_dynamic_table_new_name [id=f4d9011826fe438984b268a77313e24e]) successfully deleted
I20240503 01:59:43.792212 12405 tablet_bootstrap.cc:654] T 9e21d082e40540d4bb908fc9e90055c9 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:43.791311 6993 tablet_service.cc:1467] Processing CreateTablet for tablet aaeebd4c884445ba9f44ac9bfcb9c7ba (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=9c9f7f12e036416d8388508bfb61c799]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:43.792589 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaeebd4c884445ba9f44ac9bfcb9c7ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.792714 9342 tablet_service.cc:1467] Processing CreateTablet for tablet aaeebd4c884445ba9f44ac9bfcb9c7ba (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=9c9f7f12e036416d8388508bfb61c799]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:43.792774 12401 tablet_replica.cc:330] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a: stopping tablet replica
I20240503 01:59:43.792932 12401 raft_consensus.cc:2229] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:43.792976 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaeebd4c884445ba9f44ac9bfcb9c7ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.793152 12401 raft_consensus.cc:2258] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:43.784690 9736 tablet_service.cc:1467] Processing CreateTablet for tablet aaeebd4c884445ba9f44ac9bfcb9c7ba (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=9c9f7f12e036416d8388508bfb61c799]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:43.793465 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaeebd4c884445ba9f44ac9bfcb9c7ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.790503 9737 tablet_service.cc:1467] Processing CreateTablet for tablet 9e21d082e40540d4bb908fc9e90055c9 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=9c9f7f12e036416d8388508bfb61c799]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:43.795990 9737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e21d082e40540d4bb908fc9e90055c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.796250 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet d2fbadb911aa4bd7b4db021860c202d3 (table rebalance_test_table_5 [id=eca57da7c1764bd9a55819692641dac8]) successfully deleted
I20240503 01:59:43.796654 12411 tablet_bootstrap.cc:492] T aaeebd4c884445ba9f44ac9bfcb9c7ba P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:43.797603 12411 tablet_bootstrap.cc:654] T aaeebd4c884445ba9f44ac9bfcb9c7ba P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:43.798054 12404 tablet_bootstrap.cc:492] T a1879ffdfd2e4b26b894542d2f13a7c7 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:43.798069 12406 tablet_bootstrap.cc:492] T 9e21d082e40540d4bb908fc9e90055c9 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:43.798161 12403 raft_consensus.cc:2229] T f68d6cd3a9bd4b67b5fe23af57878b81 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:43.798352 12403 raft_consensus.cc:2258] T f68d6cd3a9bd4b67b5fe23af57878b81 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:43.799153 12406 tablet_bootstrap.cc:654] T 9e21d082e40540d4bb908fc9e90055c9 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:43.800065 12401 ts_tablet_manager.cc:1875] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:43.800293 12403 ts_tablet_manager.cc:1875] T f68d6cd3a9bd4b67b5fe23af57878b81 P db1a0000aaa346f79c0aa4b6c5105191: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:43.800860 12404 tablet_bootstrap.cc:654] T a1879ffdfd2e4b26b894542d2f13a7c7 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:43.800980 12411 tablet_bootstrap.cc:492] T aaeebd4c884445ba9f44ac9bfcb9c7ba P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:43.801128 12411 ts_tablet_manager.cc:1367] T aaeebd4c884445ba9f44ac9bfcb9c7ba P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:43.802093 12405 tablet_bootstrap.cc:492] T 9e21d082e40540d4bb908fc9e90055c9 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:43.802191 12405 ts_tablet_manager.cc:1367] T 9e21d082e40540d4bb908fc9e90055c9 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 01:59:43.802198 12404 tablet_bootstrap.cc:492] T a1879ffdfd2e4b26b894542d2f13a7c7 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:43.802271 12404 ts_tablet_manager.cc:1367] T a1879ffdfd2e4b26b894542d2f13a7c7 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:43.802829 12404 raft_consensus.cc:348] T a1879ffdfd2e4b26b894542d2f13a7c7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:43.802910 12405 raft_consensus.cc:348] T 9e21d082e40540d4bb908fc9e90055c9 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:43.803015 12405 raft_consensus.cc:374] T 9e21d082e40540d4bb908fc9e90055c9 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:43.803043 12405 raft_consensus.cc:729] T 9e21d082e40540d4bb908fc9e90055c9 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.803187 12404 raft_consensus.cc:374] T a1879ffdfd2e4b26b894542d2f13a7c7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:43.803176 12405 consensus_queue.cc:260] T 9e21d082e40540d4bb908fc9e90055c9 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:43.803279 12404 raft_consensus.cc:729] T a1879ffdfd2e4b26b894542d2f13a7c7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.803436 12404 consensus_queue.cc:260] T a1879ffdfd2e4b26b894542d2f13a7c7 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:43.803508 12405 ts_tablet_manager.cc:1398] T 9e21d082e40540d4bb908fc9e90055c9 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:43.803709 12405 tablet_bootstrap.cc:492] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:43.804404 12404 ts_tablet_manager.cc:1398] T a1879ffdfd2e4b26b894542d2f13a7c7 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:43.804672 12405 tablet_bootstrap.cc:654] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:43.807837 12405 tablet_bootstrap.cc:492] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:43.808264 12411 raft_consensus.cc:348] T aaeebd4c884445ba9f44ac9bfcb9c7ba P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:43.809249 12406 tablet_bootstrap.cc:492] T 9e21d082e40540d4bb908fc9e90055c9 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:43.809392 12406 ts_tablet_manager.cc:1367] T 9e21d082e40540d4bb908fc9e90055c9 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 01:59:43.810168 12406 raft_consensus.cc:348] T 9e21d082e40540d4bb908fc9e90055c9 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:43.810356 12406 raft_consensus.cc:374] T 9e21d082e40540d4bb908fc9e90055c9 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:43.810446 12406 raft_consensus.cc:729] T 9e21d082e40540d4bb908fc9e90055c9 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.811928 12405 ts_tablet_manager.cc:1367] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 01:59:43.812381 12406 consensus_queue.cc:260] T 9e21d082e40540d4bb908fc9e90055c9 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:43.812502 12405 raft_consensus.cc:348] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:43.813094 12411 raft_consensus.cc:374] T aaeebd4c884445ba9f44ac9bfcb9c7ba P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:43.813199 12411 raft_consensus.cc:729] T aaeebd4c884445ba9f44ac9bfcb9c7ba P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.813378 12411 consensus_queue.cc:260] T aaeebd4c884445ba9f44ac9bfcb9c7ba P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:43.813704 12406 ts_tablet_manager.cc:1398] T 9e21d082e40540d4bb908fc9e90055c9 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:43.813947 12406 tablet_bootstrap.cc:492] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:43.815443 12411 ts_tablet_manager.cc:1398] T aaeebd4c884445ba9f44ac9bfcb9c7ba P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240503 01:59:43.815591 12405 raft_consensus.cc:374] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:43.815687 12405 raft_consensus.cc:729] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.816129 12406 tablet_bootstrap.cc:654] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:43.818169 12406 tablet_bootstrap.cc:492] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:43.818259 12406 ts_tablet_manager.cc:1367] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:43.818790 12406 raft_consensus.cc:348] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:43.818902 12406 raft_consensus.cc:374] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:43.818940 12406 raft_consensus.cc:729] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.819084 12406 consensus_queue.cc:260] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:43.819164 12407 tablet_bootstrap.cc:492] T a1879ffdfd2e4b26b894542d2f13a7c7 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:43.819339 12411 tablet_bootstrap.cc:492] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:43.819377 12406 ts_tablet_manager.cc:1398] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:43.820328 12407 tablet_bootstrap.cc:654] T a1879ffdfd2e4b26b894542d2f13a7c7 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:43.817925 12405 consensus_queue.cc:260] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:43.821964 12410 tablet_bootstrap.cc:492] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:43.822185 12407 tablet_bootstrap.cc:492] T a1879ffdfd2e4b26b894542d2f13a7c7 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:43.822693 12405 ts_tablet_manager.cc:1398] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240503 01:59:43.822999 12410 tablet_bootstrap.cc:654] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:43.823645 12403 ts_tablet_manager.cc:1888] T f68d6cd3a9bd4b67b5fe23af57878b81 P db1a0000aaa346f79c0aa4b6c5105191: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:43.824381 12401 ts_tablet_manager.cc:1888] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:43.822841 12407 ts_tablet_manager.cc:1367] T a1879ffdfd2e4b26b894542d2f13a7c7 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:43.824539 12403 log.cc:1196] T f68d6cd3a9bd4b67b5fe23af57878b81 P db1a0000aaa346f79c0aa4b6c5105191: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/f68d6cd3a9bd4b67b5fe23af57878b81
I20240503 01:59:43.825026 12403 ts_tablet_manager.cc:1909] T f68d6cd3a9bd4b67b5fe23af57878b81 P db1a0000aaa346f79c0aa4b6c5105191: Deleting consensus metadata
I20240503 01:59:43.825145 12401 log.cc:1196] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/f68d6cd3a9bd4b67b5fe23af57878b81
I20240503 01:59:43.825130 12407 raft_consensus.cc:348] T a1879ffdfd2e4b26b894542d2f13a7c7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:43.825417 12407 raft_consensus.cc:374] T a1879ffdfd2e4b26b894542d2f13a7c7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:43.825505 12407 raft_consensus.cc:729] T a1879ffdfd2e4b26b894542d2f13a7c7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.825732 12407 consensus_queue.cc:260] T a1879ffdfd2e4b26b894542d2f13a7c7 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:43.826462 6726 catalog_manager.cc:4830] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): tablet f68d6cd3a9bd4b67b5fe23af57878b81 (table rebalancer_dynamic_table_new_name [id=f4d9011826fe438984b268a77313e24e]) successfully deleted
I20240503 01:59:43.826881 12401 ts_tablet_manager.cc:1909] T f68d6cd3a9bd4b67b5fe23af57878b81 P bb5364d730cb4489a0bfc6c270b7450a: Deleting consensus metadata
I20240503 01:59:43.827704 12410 tablet_bootstrap.cc:492] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:43.827808 12410 ts_tablet_manager.cc:1367] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:43.827947 12411 tablet_bootstrap.cc:654] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:43.828250 12410 raft_consensus.cc:348] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:43.828349 12410 raft_consensus.cc:374] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:43.828384 12410 raft_consensus.cc:729] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.829131 12410 consensus_queue.cc:260] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:43.829178 12411 tablet_bootstrap.cc:492] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:43.829278 12411 ts_tablet_manager.cc:1367] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 01:59:43.829520 12410 ts_tablet_manager.cc:1398] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:43.829598 6725 catalog_manager.cc:4830] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): tablet f68d6cd3a9bd4b67b5fe23af57878b81 (table rebalancer_dynamic_table_new_name [id=f4d9011826fe438984b268a77313e24e]) successfully deleted
I20240503 01:59:43.830875 12411 raft_consensus.cc:348] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:43.831017 12411 raft_consensus.cc:374] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:43.831000 12407 ts_tablet_manager.cc:1398] T a1879ffdfd2e4b26b894542d2f13a7c7 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 01:59:43.831082 12411 raft_consensus.cc:729] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.831292 12411 consensus_queue.cc:260] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:43.832247 12411 ts_tablet_manager.cc:1398] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:43.835068 12307 raft_consensus.cc:482] T a1879ffdfd2e4b26b894542d2f13a7c7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:43.835186 12307 raft_consensus.cc:504] T a1879ffdfd2e4b26b894542d2f13a7c7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:43.835739 12307 leader_election.cc:288] T a1879ffdfd2e4b26b894542d2f13a7c7 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:43.841313 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1879ffdfd2e4b26b894542d2f13a7c7" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:43.841492 6882 raft_consensus.cc:2454] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 01:59:43.841888 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1879ffdfd2e4b26b894542d2f13a7c7" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:43.842017 9494 raft_consensus.cc:2454] T a1879ffdfd2e4b26b894542d2f13a7c7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 01:59:43.842310 9559 leader_election.cc:302] T a1879ffdfd2e4b26b894542d2f13a7c7 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:43.842478 12307 raft_consensus.cc:2781] T a1879ffdfd2e4b26b894542d2f13a7c7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:43.842556 12307 raft_consensus.cc:482] T a1879ffdfd2e4b26b894542d2f13a7c7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:43.842597 12307 raft_consensus.cc:3037] T a1879ffdfd2e4b26b894542d2f13a7c7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:43.843447 12307 raft_consensus.cc:504] T a1879ffdfd2e4b26b894542d2f13a7c7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:43.843863 12307 leader_election.cc:288] T a1879ffdfd2e4b26b894542d2f13a7c7 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Requested vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:43.844220 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1879ffdfd2e4b26b894542d2f13a7c7" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:43.844231 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1879ffdfd2e4b26b894542d2f13a7c7" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:43.844350 6882 raft_consensus.cc:3037] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:43.844357 9494 raft_consensus.cc:3037] T a1879ffdfd2e4b26b894542d2f13a7c7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:43.845327 9494 raft_consensus.cc:2454] T a1879ffdfd2e4b26b894542d2f13a7c7 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 01:59:43.845327 6882 raft_consensus.cc:2454] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 01:59:43.845649 9559 leader_election.cc:302] T a1879ffdfd2e4b26b894542d2f13a7c7 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:43.845885 12307 raft_consensus.cc:2781] T a1879ffdfd2e4b26b894542d2f13a7c7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:43.845968 12307 raft_consensus.cc:686] T a1879ffdfd2e4b26b894542d2f13a7c7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 LEADER]: Becoming Leader. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Running, Role: LEADER
I20240503 01:59:43.846124 12307 consensus_queue.cc:237] T a1879ffdfd2e4b26b894542d2f13a7c7 P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:43.848517 6732 catalog_manager.cc:5484] T a1879ffdfd2e4b26b894542d2f13a7c7 P b7062fbbc2b54fff9e99ec4ea5562c76 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:43.880772 12321 raft_consensus.cc:482] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:43.880909 12321 raft_consensus.cc:504] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:43.881358 12321 leader_election.cc:288] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:43.881981 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaeebd4c884445ba9f44ac9bfcb9c7ba" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:43.882149 9363 raft_consensus.cc:2454] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:43.882117 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaeebd4c884445ba9f44ac9bfcb9c7ba" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:43.882400 9755 raft_consensus.cc:2454] T aaeebd4c884445ba9f44ac9bfcb9c7ba P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:43.882761 6949 leader_election.cc:302] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:43.882982 12321 raft_consensus.cc:2781] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:43.883039 12321 raft_consensus.cc:482] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:43.883080 12321 raft_consensus.cc:3037] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:43.883594 6740 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:43.884187 12321 raft_consensus.cc:504] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:43.884580 12321 leader_election.cc:288] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:43.884999 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaeebd4c884445ba9f44ac9bfcb9c7ba" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:43.885162 9755 raft_consensus.cc:3037] T aaeebd4c884445ba9f44ac9bfcb9c7ba P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:43.885169 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaeebd4c884445ba9f44ac9bfcb9c7ba" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:43.885278 9363 raft_consensus.cc:3037] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:43.886286 9363 raft_consensus.cc:2454] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:43.886283 9755 raft_consensus.cc:2454] T aaeebd4c884445ba9f44ac9bfcb9c7ba P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:43.886607 6949 leader_election.cc:302] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:43.887032 12321 raft_consensus.cc:2781] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:43.887132 12321 raft_consensus.cc:686] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:43.887276 12321 consensus_queue.cc:237] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:43.889564 6740 catalog_manager.cc:5484] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:43.889940 9473 tablet_service.cc:1467] Processing CreateTablet for tablet 29637fcc22fe42208e9b982ef39d5d33 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=27f16aef77d342f1b0dc856aa0612d60]), partition=
I20240503 01:59:43.890210 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29637fcc22fe42208e9b982ef39d5d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.891216 9604 tablet_service.cc:1467] Processing CreateTablet for tablet 29637fcc22fe42208e9b982ef39d5d33 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=27f16aef77d342f1b0dc856aa0612d60]), partition=
I20240503 01:59:43.891465 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29637fcc22fe42208e9b982ef39d5d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.892537 12404 tablet_bootstrap.cc:492] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:43.893487 12404 tablet_bootstrap.cc:654] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:43.893518 12407 tablet_bootstrap.cc:492] T 29637fcc22fe42208e9b982ef39d5d33 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:43.894186 9737 tablet_service.cc:1467] Processing CreateTablet for tablet 29637fcc22fe42208e9b982ef39d5d33 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=27f16aef77d342f1b0dc856aa0612d60]), partition=
I20240503 01:59:43.894446 9737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29637fcc22fe42208e9b982ef39d5d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:43.894698 12407 tablet_bootstrap.cc:654] T 29637fcc22fe42208e9b982ef39d5d33 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:43.895752 12404 tablet_bootstrap.cc:492] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:43.895846 12404 ts_tablet_manager.cc:1367] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20240503 01:59:43.896315 12404 raft_consensus.cc:348] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:43.896425 12404 raft_consensus.cc:374] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:43.896456 12404 raft_consensus.cc:729] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.896608 12411 tablet_bootstrap.cc:492] T 29637fcc22fe42208e9b982ef39d5d33 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:43.896786 12404 consensus_queue.cc:260] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:43.897547 12411 tablet_bootstrap.cc:654] T 29637fcc22fe42208e9b982ef39d5d33 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:43.898380 12404 ts_tablet_manager.cc:1398] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:43.899570 12407 tablet_bootstrap.cc:492] T 29637fcc22fe42208e9b982ef39d5d33 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:43.899678 12407 ts_tablet_manager.cc:1367] T 29637fcc22fe42208e9b982ef39d5d33 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:43.899890 12411 tablet_bootstrap.cc:492] T 29637fcc22fe42208e9b982ef39d5d33 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:43.899963 12411 ts_tablet_manager.cc:1367] T 29637fcc22fe42208e9b982ef39d5d33 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:43.900084 12407 raft_consensus.cc:348] T 29637fcc22fe42208e9b982ef39d5d33 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:43.900173 12407 raft_consensus.cc:374] T 29637fcc22fe42208e9b982ef39d5d33 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:43.900193 12407 raft_consensus.cc:729] T 29637fcc22fe42208e9b982ef39d5d33 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.900305 12407 consensus_queue.cc:260] T 29637fcc22fe42208e9b982ef39d5d33 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:43.900388 12411 raft_consensus.cc:348] T 29637fcc22fe42208e9b982ef39d5d33 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:43.900481 12411 raft_consensus.cc:374] T 29637fcc22fe42208e9b982ef39d5d33 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:43.900508 12411 raft_consensus.cc:729] T 29637fcc22fe42208e9b982ef39d5d33 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:43.900583 12407 ts_tablet_manager.cc:1398] T 29637fcc22fe42208e9b982ef39d5d33 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:43.900609 12411 consensus_queue.cc:260] T 29637fcc22fe42208e9b982ef39d5d33 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:43.900841 12411 ts_tablet_manager.cc:1398] T 29637fcc22fe42208e9b982ef39d5d33 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:43.934514 12070 raft_consensus.cc:482] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:43.934795 12070 raft_consensus.cc:504] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:43.935354 12070 leader_election.cc:288] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:43.935775 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637fcc22fe42208e9b982ef39d5d33" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:43.935855 9626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637fcc22fe42208e9b982ef39d5d33" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:43.935930 9755 raft_consensus.cc:2454] T 29637fcc22fe42208e9b982ef39d5d33 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:43.936028 9626 raft_consensus.cc:2454] T 29637fcc22fe42208e9b982ef39d5d33 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:43.936271 9428 leader_election.cc:302] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:43.936442 12070 raft_consensus.cc:2781] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:43.936506 12070 raft_consensus.cc:482] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:43.936547 12070 raft_consensus.cc:3037] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:43.937549 12070 raft_consensus.cc:504] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:43.937978 12070 leader_election.cc:288] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Requested vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:43.938254 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637fcc22fe42208e9b982ef39d5d33" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:43.938378 9755 raft_consensus.cc:3037] T 29637fcc22fe42208e9b982ef39d5d33 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:43.938362 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29637fcc22fe42208e9b982ef39d5d33" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:43.938485 9625 raft_consensus.cc:3037] T 29637fcc22fe42208e9b982ef39d5d33 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:43.939257 9755 raft_consensus.cc:2454] T 29637fcc22fe42208e9b982ef39d5d33 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:43.939287 9625 raft_consensus.cc:2454] T 29637fcc22fe42208e9b982ef39d5d33 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:43.939572 9428 leader_election.cc:302] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:43.940004 12070 raft_consensus.cc:2781] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:43.940176 12070 raft_consensus.cc:686] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Becoming Leader. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Running, Role: LEADER
I20240503 01:59:43.940371 12070 consensus_queue.cc:237] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:43.942255 6740 catalog_manager.cc:5484] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: term changed from 0 to 1, leader changed from <none> to bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:44.054579 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:44.058677 9625 raft_consensus.cc:1261] T 29637fcc22fe42208e9b982ef39d5d33 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:44.058687 9755 raft_consensus.cc:1261] T 29637fcc22fe42208e9b982ef39d5d33 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:44.059145 12457 consensus_queue.cc:1035] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:44.059298 12070 consensus_queue.cc:1035] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:44.061970 11986 tablet.cc:1722] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:44.062335 11986 tablet.cc:1620] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:44.063176 6737 catalog_manager.cc:5576] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087) does not have the latest schema for tablet 29637fcc22fe42208e9b982ef39d5d33 (table rebalancer_dynamic_table [id=27f16aef77d342f1b0dc856aa0612d60]). Expected version 1 got 0
I20240503 01:59:44.063341 12292 tablet.cc:1722] T 29637fcc22fe42208e9b982ef39d5d33 P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:44.063659 12292 tablet.cc:1620] T 29637fcc22fe42208e9b982ef39d5d33 P b7062fbbc2b54fff9e99ec4ea5562c76: MemRowSet was empty: no flush needed.
I20240503 01:59:44.064255 6740 catalog_manager.cc:5576] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053) does not have the latest schema for tablet 29637fcc22fe42208e9b982ef39d5d33 (table rebalancer_dynamic_table [id=27f16aef77d342f1b0dc856aa0612d60]). Expected version 1 got 0
I20240503 01:59:44.065753 12294 tablet.cc:1722] T 29637fcc22fe42208e9b982ef39d5d33 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:44.066046 12294 tablet.cc:1620] T 29637fcc22fe42208e9b982ef39d5d33 P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:44.069715 6737 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=27f16aef77d342f1b0dc856aa0612d60] alter complete (version 1)
I20240503 01:59:44.175763 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:44.181849 11986 tablet.cc:1722] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:44.182464 11986 tablet.cc:1620] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:44.183013 12292 tablet.cc:1722] T 29637fcc22fe42208e9b982ef39d5d33 P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:44.183147 12294 tablet.cc:1722] T 29637fcc22fe42208e9b982ef39d5d33 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:44.183362 12292 tablet.cc:1620] T 29637fcc22fe42208e9b982ef39d5d33 P b7062fbbc2b54fff9e99ec4ea5562c76: MemRowSet was empty: no flush needed.
I20240503 01:59:44.183434 12294 tablet.cc:1620] T 29637fcc22fe42208e9b982ef39d5d33 P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:44.188702 6734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=27f16aef77d342f1b0dc856aa0612d60] alter complete (version 2)
I20240503 01:59:44.210295 12011 raft_consensus.cc:482] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:44.210458 12011 raft_consensus.cc:504] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:44.211066 12011 leader_election.cc:288] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:44.211324 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e21d082e40540d4bb908fc9e90055c9" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:44.211473 9363 raft_consensus.cc:2454] T 9e21d082e40540d4bb908fc9e90055c9 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 01:59:44.211403 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e21d082e40540d4bb908fc9e90055c9" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:44.211591 6882 raft_consensus.cc:2454] T 9e21d082e40540d4bb908fc9e90055c9 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 01:59:44.211781 9692 leader_election.cc:302] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:44.212002 12011 raft_consensus.cc:2781] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:44.212064 12011 raft_consensus.cc:482] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:44.212105 12011 raft_consensus.cc:3037] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:44.213173 12011 raft_consensus.cc:504] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:44.213541 12011 leader_election.cc:288] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Requested vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:44.213833 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e21d082e40540d4bb908fc9e90055c9" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:44.213864 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e21d082e40540d4bb908fc9e90055c9" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:44.213944 6882 raft_consensus.cc:3037] T 9e21d082e40540d4bb908fc9e90055c9 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:44.213999 9363 raft_consensus.cc:3037] T 9e21d082e40540d4bb908fc9e90055c9 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:44.214833 9363 raft_consensus.cc:2454] T 9e21d082e40540d4bb908fc9e90055c9 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:44.214838 6882 raft_consensus.cc:2454] T 9e21d082e40540d4bb908fc9e90055c9 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:44.215137 9693 leader_election.cc:302] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:44.215312 12011 raft_consensus.cc:2781] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:44.215497 12011 raft_consensus.cc:686] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Becoming Leader. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Running, Role: LEADER
I20240503 01:59:44.215658 12011 consensus_queue.cc:237] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:44.217767 6734 catalog_manager.cc:5484] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: term changed from 0 to 1, leader changed from <none> to db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:44.247004 6993 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:44.248191 6861 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:44.252382 9342 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:44.255064 9473 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:44.259882 7124 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:44.261731 9737 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:44.263444 7256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:44.271631 9604 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:44.288498 12321 consensus_queue.cc:1035] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:44.295233 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:44.303721 11986 tablet.cc:1722] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:44.309530 10855 consensus_queue.cc:1035] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:44.311215 12292 tablet.cc:1722] T 29637fcc22fe42208e9b982ef39d5d33 P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:44.313505 6737 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=27f16aef77d342f1b0dc856aa0612d60] alter complete (version 3)
I20240503 01:59:44.320757 12294 tablet.cc:1722] T 29637fcc22fe42208e9b982ef39d5d33 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:44.362643 6882 consensus_queue.cc:237] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:44.364223 9363 raft_consensus.cc:1261] T 72e090a6c18e4a13bbb343eb40a07694 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:44.364367 7145 raft_consensus.cc:1261] T 72e090a6c18e4a13bbb343eb40a07694 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:44.364734 10668 consensus_queue.cc:1035] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:44.364940 11520 consensus_queue.cc:1035] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:44.366874 12504 raft_consensus.cc:2932] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:44.367278 7145 raft_consensus.cc:2932] T 72e090a6c18e4a13bbb343eb40a07694 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:44.369225 9363 raft_consensus.cc:2932] T 72e090a6c18e4a13bbb343eb40a07694 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:44.372368 6734 catalog_manager.cc:5484] T 72e090a6c18e4a13bbb343eb40a07694 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index 4 to 5, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:44.376060 7256 tablet_service.cc:1514] Processing DeleteTablet for tablet 72e090a6c18e4a13bbb343eb40a07694 with delete_type TABLET_DATA_TOMBSTONED (TS 1d0e738c34af40fa9950c6f086619b3a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:44.376698 12510 tablet_replica.cc:330] T 72e090a6c18e4a13bbb343eb40a07694 P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:44.376875 12510 raft_consensus.cc:2229] T 72e090a6c18e4a13bbb343eb40a07694 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:44.377151 12510 raft_consensus.cc:2258] T 72e090a6c18e4a13bbb343eb40a07694 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:44.377795 12510 ts_tablet_manager.cc:1875] T 72e090a6c18e4a13bbb343eb40a07694 P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:44.380707 12510 ts_tablet_manager.cc:1888] T 72e090a6c18e4a13bbb343eb40a07694 P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 01:59:44.380810 12510 log.cc:1196] T 72e090a6c18e4a13bbb343eb40a07694 P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/72e090a6c18e4a13bbb343eb40a07694
I20240503 01:59:44.381453 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet 72e090a6c18e4a13bbb343eb40a07694 (table rebalance_test_table_9 [id=f050ea79f4334dcda01dce5d3e804176]) successfully deleted
I20240503 01:59:44.418426 6734 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:44.418606 6734 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:44.421157 6734 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 29637fcc22fe42208e9b982ef39d5d33
I20240503 01:59:44.423532 9604 tablet_service.cc:1514] Processing DeleteTablet for tablet 29637fcc22fe42208e9b982ef39d5d33 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:44 UTC) from {username='slave'} at 127.0.0.1:58562
I20240503 01:59:44.423715 9737 tablet_service.cc:1514] Processing DeleteTablet for tablet 29637fcc22fe42208e9b982ef39d5d33 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:44 UTC) from {username='slave'} at 127.0.0.1:48152
I20240503 01:59:44.423890 12547 tablet_replica.cc:330] T 29637fcc22fe42208e9b982ef39d5d33 P b7062fbbc2b54fff9e99ec4ea5562c76: stopping tablet replica
I20240503 01:59:44.424050 12548 tablet_replica.cc:330] T 29637fcc22fe42208e9b982ef39d5d33 P db1a0000aaa346f79c0aa4b6c5105191: stopping tablet replica
I20240503 01:59:44.424083 9473 tablet_service.cc:1514] Processing DeleteTablet for tablet 29637fcc22fe42208e9b982ef39d5d33 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:44 UTC) from {username='slave'} at 127.0.0.1:54678
I20240503 01:59:44.424150 12547 raft_consensus.cc:2229] T 29637fcc22fe42208e9b982ef39d5d33 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:44.424214 12548 raft_consensus.cc:2229] T 29637fcc22fe42208e9b982ef39d5d33 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:44.424290 12547 raft_consensus.cc:2258] T 29637fcc22fe42208e9b982ef39d5d33 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:44.424324 12548 raft_consensus.cc:2258] T 29637fcc22fe42208e9b982ef39d5d33 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:44.424880 12549 tablet_replica.cc:330] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a: stopping tablet replica
I20240503 01:59:44.424968 12548 ts_tablet_manager.cc:1875] T 29637fcc22fe42208e9b982ef39d5d33 P db1a0000aaa346f79c0aa4b6c5105191: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:44.427424 12547 ts_tablet_manager.cc:1875] T 29637fcc22fe42208e9b982ef39d5d33 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:44.425072 12549 raft_consensus.cc:2229] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:44.427984 12549 raft_consensus.cc:2258] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:44.429158 12548 ts_tablet_manager.cc:1888] T 29637fcc22fe42208e9b982ef39d5d33 P db1a0000aaa346f79c0aa4b6c5105191: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:44.429322 12548 log.cc:1196] T 29637fcc22fe42208e9b982ef39d5d33 P db1a0000aaa346f79c0aa4b6c5105191: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/29637fcc22fe42208e9b982ef39d5d33
I20240503 01:59:44.430263 12548 ts_tablet_manager.cc:1909] T 29637fcc22fe42208e9b982ef39d5d33 P db1a0000aaa346f79c0aa4b6c5105191: Deleting consensus metadata
I20240503 01:59:44.430828 12547 ts_tablet_manager.cc:1888] T 29637fcc22fe42208e9b982ef39d5d33 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:44.430975 12547 log.cc:1196] T 29637fcc22fe42208e9b982ef39d5d33 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/29637fcc22fe42208e9b982ef39d5d33
I20240503 01:59:44.431375 12547 ts_tablet_manager.cc:1909] T 29637fcc22fe42208e9b982ef39d5d33 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting consensus metadata
I20240503 01:59:44.431423 6726 catalog_manager.cc:4830] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): tablet 29637fcc22fe42208e9b982ef39d5d33 (table rebalancer_dynamic_table_new_name [id=27f16aef77d342f1b0dc856aa0612d60]) successfully deleted
I20240503 01:59:44.432297 12549 ts_tablet_manager.cc:1875] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:44.432425 6727 catalog_manager.cc:4830] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): tablet 29637fcc22fe42208e9b982ef39d5d33 (table rebalancer_dynamic_table_new_name [id=27f16aef77d342f1b0dc856aa0612d60]) successfully deleted
I20240503 01:59:44.434958 12549 ts_tablet_manager.cc:1888] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:44.435058 12549 log.cc:1196] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/29637fcc22fe42208e9b982ef39d5d33
I20240503 01:59:44.435331 12549 ts_tablet_manager.cc:1909] T 29637fcc22fe42208e9b982ef39d5d33 P bb5364d730cb4489a0bfc6c270b7450a: Deleting consensus metadata
I20240503 01:59:44.436165 6725 catalog_manager.cc:4830] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): tablet 29637fcc22fe42208e9b982ef39d5d33 (table rebalancer_dynamic_table_new_name [id=27f16aef77d342f1b0dc856aa0612d60]) successfully deleted
I20240503 01:59:44.445564 11786 consensus_queue.cc:1035] T a1879ffdfd2e4b26b894542d2f13a7c7 P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:44.446645 11786 consensus_queue.cc:1035] T a1879ffdfd2e4b26b894542d2f13a7c7 P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:44.524322 6738 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:44.537364 6993 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:44.538146 9604 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:44.540946 9342 tablet_service.cc:1467] Processing CreateTablet for tablet eb9d11716a3341589cd827cad3b88fa5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=608ae4072a8041c5945a6202e671de59]), partition=
I20240503 01:59:44.541234 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb9d11716a3341589cd827cad3b88fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:44.542783 9604 tablet_service.cc:1467] Processing CreateTablet for tablet eb9d11716a3341589cd827cad3b88fa5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=608ae4072a8041c5945a6202e671de59]), partition=
I20240503 01:59:44.543042 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb9d11716a3341589cd827cad3b88fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:44.543884 7124 tablet_service.cc:1467] Processing CreateTablet for tablet eb9d11716a3341589cd827cad3b88fa5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=608ae4072a8041c5945a6202e671de59]), partition=
I20240503 01:59:44.544144 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb9d11716a3341589cd827cad3b88fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:44.548030 7124 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:44.548389 12608 tablet_bootstrap.cc:492] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:44.549932 12606 tablet_bootstrap.cc:492] T eb9d11716a3341589cd827cad3b88fa5 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:44.552075 12608 tablet_bootstrap.cc:654] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:44.553139 12606 tablet_bootstrap.cc:654] T eb9d11716a3341589cd827cad3b88fa5 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:44.555511 9737 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:44.567317 9342 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:44.573249 12606 tablet_bootstrap.cc:492] T eb9d11716a3341589cd827cad3b88fa5 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:44.573467 12606 ts_tablet_manager.cc:1367] T eb9d11716a3341589cd827cad3b88fa5 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240503 01:59:44.575271 12606 raft_consensus.cc:348] T eb9d11716a3341589cd827cad3b88fa5 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:44.576021 12606 raft_consensus.cc:374] T eb9d11716a3341589cd827cad3b88fa5 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:44.576118 12606 raft_consensus.cc:729] T eb9d11716a3341589cd827cad3b88fa5 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:44.576304 12606 consensus_queue.cc:260] T eb9d11716a3341589cd827cad3b88fa5 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:44.576762 12606 ts_tablet_manager.cc:1398] T eb9d11716a3341589cd827cad3b88fa5 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:44.578995 6861 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:44.584486 9473 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:44.585032 7256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:44.588673 12607 tablet_bootstrap.cc:492] T eb9d11716a3341589cd827cad3b88fa5 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:44.591740 12607 tablet_bootstrap.cc:654] T eb9d11716a3341589cd827cad3b88fa5 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:44.593775 12607 tablet_bootstrap.cc:492] T eb9d11716a3341589cd827cad3b88fa5 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:44.593966 12607 ts_tablet_manager.cc:1367] T eb9d11716a3341589cd827cad3b88fa5 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 01:59:44.594586 12607 raft_consensus.cc:348] T eb9d11716a3341589cd827cad3b88fa5 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:44.594782 12607 raft_consensus.cc:374] T eb9d11716a3341589cd827cad3b88fa5 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:44.594851 12607 raft_consensus.cc:729] T eb9d11716a3341589cd827cad3b88fa5 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:44.595063 12607 consensus_queue.cc:260] T eb9d11716a3341589cd827cad3b88fa5 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:44.595496 12607 ts_tablet_manager.cc:1398] T eb9d11716a3341589cd827cad3b88fa5 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:44.596735 12608 tablet_bootstrap.cc:492] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:44.596841 12608 ts_tablet_manager.cc:1367] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.000s
I20240503 01:59:44.597321 12608 raft_consensus.cc:348] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:44.597422 12608 raft_consensus.cc:374] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:44.597448 12608 raft_consensus.cc:729] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:44.597596 12608 consensus_queue.cc:260] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:44.598003 12608 ts_tablet_manager.cc:1398] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:44.655372 7275 consensus_queue.cc:237] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:44.656677 9626 raft_consensus.cc:1261] T c029253a832e46a48ae16691fc044502 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:44.656936 6882 raft_consensus.cc:1261] T c029253a832e46a48ae16691fc044502 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:44.657104 12628 consensus_queue.cc:1035] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:44.657366 12366 consensus_queue.cc:1035] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:44.659032 12628 raft_consensus.cc:2932] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:44.659464 6882 raft_consensus.cc:2932] T c029253a832e46a48ae16691fc044502 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:44.659878 9626 raft_consensus.cc:2932] T c029253a832e46a48ae16691fc044502 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:44.662909 6739 catalog_manager.cc:5484] T c029253a832e46a48ae16691fc044502 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:44.666029 7124 tablet_service.cc:1514] Processing DeleteTablet for tablet c029253a832e46a48ae16691fc044502 with delete_type TABLET_DATA_TOMBSTONED (TS c688f3df56ac4b6d9b0c30fa4cc8ca2a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:44.666999 12631 tablet_replica.cc:330] T c029253a832e46a48ae16691fc044502 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:44.668061 12631 raft_consensus.cc:2229] T c029253a832e46a48ae16691fc044502 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:44.668334 12631 raft_consensus.cc:2258] T c029253a832e46a48ae16691fc044502 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:44.668846 12631 ts_tablet_manager.cc:1875] T c029253a832e46a48ae16691fc044502 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:44.671234 12631 ts_tablet_manager.cc:1888] T c029253a832e46a48ae16691fc044502 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 01:59:44.671310 12631 log.cc:1196] T c029253a832e46a48ae16691fc044502 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/c029253a832e46a48ae16691fc044502
I20240503 01:59:44.671849 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet c029253a832e46a48ae16691fc044502 (table rebalance_test_table_7 [id=3428bac98e0c49908450b3b51cb9d18c]) successfully deleted
I20240503 01:59:44.702591 12011 consensus_queue.cc:1035] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:44.706117 12011 consensus_queue.cc:1035] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:44.736457 12031 raft_consensus.cc:482] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:44.736603 12031 raft_consensus.cc:504] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:44.737051 12031 leader_election.cc:288] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:44.737710 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9d11716a3341589cd827cad3b88fa5" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:44.737710 9626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9d11716a3341589cd827cad3b88fa5" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:44.737859 9363 raft_consensus.cc:2454] T eb9d11716a3341589cd827cad3b88fa5 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:44.737859 9626 raft_consensus.cc:2454] T eb9d11716a3341589cd827cad3b88fa5 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:44.739053 7080 leader_election.cc:302] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:44.739677 12031 raft_consensus.cc:2781] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:44.739742 12031 raft_consensus.cc:482] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:44.739782 12031 raft_consensus.cc:3037] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:44.741029 12031 raft_consensus.cc:504] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:44.741458 12031 leader_election.cc:288] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Requested vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:44.742071 9626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9d11716a3341589cd827cad3b88fa5" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:44.742076 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9d11716a3341589cd827cad3b88fa5" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:44.742224 9626 raft_consensus.cc:3037] T eb9d11716a3341589cd827cad3b88fa5 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:44.742224 9363 raft_consensus.cc:3037] T eb9d11716a3341589cd827cad3b88fa5 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:44.743388 9626 raft_consensus.cc:2454] T eb9d11716a3341589cd827cad3b88fa5 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:44.744030 9363 raft_consensus.cc:2454] T eb9d11716a3341589cd827cad3b88fa5 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:44.744495 7080 leader_election.cc:302] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:44.744829 12031 raft_consensus.cc:2781] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:44.745076 12031 raft_consensus.cc:686] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Becoming Leader. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Running, Role: LEADER
I20240503 01:59:44.745241 12031 consensus_queue.cc:237] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:44.749516 6737 catalog_manager.cc:5484] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:44.762985 6737 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:44.768534 7256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:44.774398 7124 tablet_service.cc:1467] Processing CreateTablet for tablet c43f13a33fe94033bce9debbae6990c2 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2d05875281f54da8b4f2477aaf91b494]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:44.774398 9604 tablet_service.cc:1467] Processing CreateTablet for tablet beb175f81d9e48229466503ebf7e3c09 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2d05875281f54da8b4f2477aaf91b494]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:44.774376 9342 tablet_service.cc:1467] Processing CreateTablet for tablet beb175f81d9e48229466503ebf7e3c09 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2d05875281f54da8b4f2477aaf91b494]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:44.774765 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beb175f81d9e48229466503ebf7e3c09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:44.774803 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beb175f81d9e48229466503ebf7e3c09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:44.774938 6993 tablet_service.cc:1467] Processing CreateTablet for tablet 5ec2909a6e484f81aad7b39a374e1ef8 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2d05875281f54da8b4f2477aaf91b494]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:44.775197 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ec2909a6e484f81aad7b39a374e1ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:44.777302 12607 tablet_bootstrap.cc:492] T beb175f81d9e48229466503ebf7e3c09 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:44.777263 9341 tablet_service.cc:1467] Processing CreateTablet for tablet c43f13a33fe94033bce9debbae6990c2 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2d05875281f54da8b4f2477aaf91b494]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:44.777541 9341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c43f13a33fe94033bce9debbae6990c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:44.778575 12607 tablet_bootstrap.cc:654] T beb175f81d9e48229466503ebf7e3c09 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:44.779106 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c43f13a33fe94033bce9debbae6990c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:44.779809 12606 tablet_bootstrap.cc:492] T c43f13a33fe94033bce9debbae6990c2 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:44.782407 12690 tablet_bootstrap.cc:492] T 5ec2909a6e484f81aad7b39a374e1ef8 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:44.782773 12606 tablet_bootstrap.cc:654] T c43f13a33fe94033bce9debbae6990c2 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:44.783586 12690 tablet_bootstrap.cc:654] T 5ec2909a6e484f81aad7b39a374e1ef8 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:44.785277 9473 tablet_service.cc:1467] Processing CreateTablet for tablet beb175f81d9e48229466503ebf7e3c09 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2d05875281f54da8b4f2477aaf91b494]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:44.785589 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beb175f81d9e48229466503ebf7e3c09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:44.785537 9472 tablet_service.cc:1467] Processing CreateTablet for tablet c43f13a33fe94033bce9debbae6990c2 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2d05875281f54da8b4f2477aaf91b494]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:44.785805 9472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c43f13a33fe94033bce9debbae6990c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:44.786036 9474 tablet_service.cc:1467] Processing CreateTablet for tablet 5ec2909a6e484f81aad7b39a374e1ef8 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2d05875281f54da8b4f2477aaf91b494]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:44.786265 9474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ec2909a6e484f81aad7b39a374e1ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:44.786306 12608 tablet_bootstrap.cc:492] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:44.787351 12608 tablet_bootstrap.cc:654] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:44.790169 6993 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:44.791759 12608 tablet_bootstrap.cc:492] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:44.791862 12608 ts_tablet_manager.cc:1367] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:44.792328 12608 raft_consensus.cc:348] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:44.792443 12608 raft_consensus.cc:374] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:44.792486 12608 raft_consensus.cc:729] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:44.792595 12608 consensus_queue.cc:260] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:44.792869 12608 ts_tablet_manager.cc:1398] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:44.798813 6861 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:44.799232 12695 tablet_bootstrap.cc:492] T 5ec2909a6e484f81aad7b39a374e1ef8 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:44.800297 12695 tablet_bootstrap.cc:654] T 5ec2909a6e484f81aad7b39a374e1ef8 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:44.803103 7124 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:44.808750 12695 tablet_bootstrap.cc:492] T 5ec2909a6e484f81aad7b39a374e1ef8 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:44.808966 12695 ts_tablet_manager.cc:1367] T 5ec2909a6e484f81aad7b39a374e1ef8 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 01:59:44.809561 12695 raft_consensus.cc:348] T 5ec2909a6e484f81aad7b39a374e1ef8 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:44.809779 12695 raft_consensus.cc:374] T 5ec2909a6e484f81aad7b39a374e1ef8 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:44.809870 12695 raft_consensus.cc:729] T 5ec2909a6e484f81aad7b39a374e1ef8 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:44.810071 12695 consensus_queue.cc:260] T 5ec2909a6e484f81aad7b39a374e1ef8 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:44.810588 9737 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:44.811173 12695 ts_tablet_manager.cc:1398] T 5ec2909a6e484f81aad7b39a374e1ef8 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:44.811395 12695 tablet_bootstrap.cc:492] T beb175f81d9e48229466503ebf7e3c09 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:44.812541 12695 tablet_bootstrap.cc:654] T beb175f81d9e48229466503ebf7e3c09 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:44.813514 7256 tablet_service.cc:1467] Processing CreateTablet for tablet 5ec2909a6e484f81aad7b39a374e1ef8 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2d05875281f54da8b4f2477aaf91b494]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:44.813820 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ec2909a6e484f81aad7b39a374e1ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:44.817509 12606 tablet_bootstrap.cc:492] T c43f13a33fe94033bce9debbae6990c2 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:44.817627 12606 ts_tablet_manager.cc:1367] T c43f13a33fe94033bce9debbae6990c2 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
I20240503 01:59:44.818079 12606 raft_consensus.cc:348] T c43f13a33fe94033bce9debbae6990c2 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:44.818193 12606 raft_consensus.cc:374] T c43f13a33fe94033bce9debbae6990c2 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:44.818219 12606 raft_consensus.cc:729] T c43f13a33fe94033bce9debbae6990c2 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:44.818331 12606 consensus_queue.cc:260] T c43f13a33fe94033bce9debbae6990c2 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:44.819028 12606 ts_tablet_manager.cc:1398] T c43f13a33fe94033bce9debbae6990c2 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:44.819170 12690 tablet_bootstrap.cc:492] T 5ec2909a6e484f81aad7b39a374e1ef8 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:44.819270 12690 ts_tablet_manager.cc:1367] T 5ec2909a6e484f81aad7b39a374e1ef8 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.000s
I20240503 01:59:44.819743 12690 raft_consensus.cc:348] T 5ec2909a6e484f81aad7b39a374e1ef8 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:44.819867 12690 raft_consensus.cc:374] T 5ec2909a6e484f81aad7b39a374e1ef8 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:44.819903 12690 raft_consensus.cc:729] T 5ec2909a6e484f81aad7b39a374e1ef8 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:44.820032 12690 consensus_queue.cc:260] T 5ec2909a6e484f81aad7b39a374e1ef8 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:44.820303 12690 ts_tablet_manager.cc:1398] T 5ec2909a6e484f81aad7b39a374e1ef8 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:44.820647 12606 tablet_bootstrap.cc:492] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:44.821097 12695 tablet_bootstrap.cc:492] T beb175f81d9e48229466503ebf7e3c09 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:44.821182 12695 ts_tablet_manager.cc:1367] T beb175f81d9e48229466503ebf7e3c09 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 01:59:44.821635 12695 raft_consensus.cc:348] T beb175f81d9e48229466503ebf7e3c09 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:44.821724 12606 tablet_bootstrap.cc:654] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:44.821733 12695 raft_consensus.cc:374] T beb175f81d9e48229466503ebf7e3c09 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:44.821763 12695 raft_consensus.cc:729] T beb175f81d9e48229466503ebf7e3c09 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:44.821875 12695 consensus_queue.cc:260] T beb175f81d9e48229466503ebf7e3c09 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:44.822031 12698 tablet_bootstrap.cc:492] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:44.822145 12695 ts_tablet_manager.cc:1398] T beb175f81d9e48229466503ebf7e3c09 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:44.822288 12695 tablet_bootstrap.cc:492] T c43f13a33fe94033bce9debbae6990c2 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:44.823732 12695 tablet_bootstrap.cc:654] T c43f13a33fe94033bce9debbae6990c2 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:44.832394 12698 tablet_bootstrap.cc:654] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:44.833693 9604 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:44.838696 9342 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:44.845191 12607 tablet_bootstrap.cc:492] T beb175f81d9e48229466503ebf7e3c09 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:44.845311 12607 ts_tablet_manager.cc:1367] T beb175f81d9e48229466503ebf7e3c09 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.068s user 0.003s sys 0.000s
I20240503 01:59:44.845852 12607 raft_consensus.cc:348] T beb175f81d9e48229466503ebf7e3c09 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:44.845980 12607 raft_consensus.cc:374] T beb175f81d9e48229466503ebf7e3c09 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:44.846010 12607 raft_consensus.cc:729] T beb175f81d9e48229466503ebf7e3c09 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:44.846138 12607 consensus_queue.cc:260] T beb175f81d9e48229466503ebf7e3c09 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:44.846410 12607 ts_tablet_manager.cc:1398] T beb175f81d9e48229466503ebf7e3c09 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:44.846841 9472 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 01:59:44.856678 12606 tablet_bootstrap.cc:492] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:44.856812 12606 ts_tablet_manager.cc:1367] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.000s
I20240503 01:59:44.856870 12695 tablet_bootstrap.cc:492] T c43f13a33fe94033bce9debbae6990c2 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:44.856961 12695 ts_tablet_manager.cc:1367] T c43f13a33fe94033bce9debbae6990c2 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.000s
I20240503 01:59:44.857365 12606 raft_consensus.cc:348] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:44.857513 12606 raft_consensus.cc:374] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:44.857550 12606 raft_consensus.cc:729] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:44.857594 12695 raft_consensus.cc:348] T c43f13a33fe94033bce9debbae6990c2 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:44.857687 12695 raft_consensus.cc:374] T c43f13a33fe94033bce9debbae6990c2 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:44.857722 12695 raft_consensus.cc:729] T c43f13a33fe94033bce9debbae6990c2 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:44.857695 12606 consensus_queue.cc:260] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:44.857853 12695 consensus_queue.cc:260] T c43f13a33fe94033bce9debbae6990c2 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:44.858024 12606 ts_tablet_manager.cc:1398] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:44.858167 12695 ts_tablet_manager.cc:1398] T c43f13a33fe94033bce9debbae6990c2 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:44.867975 12698 tablet_bootstrap.cc:492] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:44.868121 12698 ts_tablet_manager.cc:1367] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.000s
I20240503 01:59:44.868577 12698 raft_consensus.cc:348] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:44.868712 12698 raft_consensus.cc:374] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:44.868774 12698 raft_consensus.cc:729] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:44.868912 12698 consensus_queue.cc:260] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:44.869171 12698 ts_tablet_manager.cc:1398] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:44.894376 12070 raft_consensus.cc:482] T 5ec2909a6e484f81aad7b39a374e1ef8 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:44.894508 12070 raft_consensus.cc:504] T 5ec2909a6e484f81aad7b39a374e1ef8 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:44.894961 12070 leader_election.cc:288] T 5ec2909a6e484f81aad7b39a374e1ef8 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:44.895434 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ec2909a6e484f81aad7b39a374e1ef8" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:44.895434 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ec2909a6e484f81aad7b39a374e1ef8" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:44.895592 7014 raft_consensus.cc:2454] T 5ec2909a6e484f81aad7b39a374e1ef8 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:44.895592 7275 raft_consensus.cc:2454] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:44.895913 9428 leader_election.cc:302] T 5ec2909a6e484f81aad7b39a374e1ef8 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:44.896090 12070 raft_consensus.cc:2781] T 5ec2909a6e484f81aad7b39a374e1ef8 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:44.896137 12070 raft_consensus.cc:482] T 5ec2909a6e484f81aad7b39a374e1ef8 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:44.896170 12070 raft_consensus.cc:3037] T 5ec2909a6e484f81aad7b39a374e1ef8 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:44.897223 12070 raft_consensus.cc:504] T 5ec2909a6e484f81aad7b39a374e1ef8 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:44.897622 12070 leader_election.cc:288] T 5ec2909a6e484f81aad7b39a374e1ef8 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Requested vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:44.898011 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ec2909a6e484f81aad7b39a374e1ef8" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:44.898141 7275 raft_consensus.cc:3037] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:44.898360 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ec2909a6e484f81aad7b39a374e1ef8" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:44.898486 7014 raft_consensus.cc:3037] T 5ec2909a6e484f81aad7b39a374e1ef8 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:44.899322 7275 raft_consensus.cc:2454] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:44.899523 7014 raft_consensus.cc:2454] T 5ec2909a6e484f81aad7b39a374e1ef8 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:44.899649 9428 leader_election.cc:302] T 5ec2909a6e484f81aad7b39a374e1ef8 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:44.899825 12070 raft_consensus.cc:2781] T 5ec2909a6e484f81aad7b39a374e1ef8 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:44.900005 12070 raft_consensus.cc:686] T 5ec2909a6e484f81aad7b39a374e1ef8 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Becoming Leader. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Running, Role: LEADER
I20240503 01:59:44.900167 12070 consensus_queue.cc:237] T 5ec2909a6e484f81aad7b39a374e1ef8 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:44.902170 6737 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:44.903277 6735 catalog_manager.cc:5484] T 5ec2909a6e484f81aad7b39a374e1ef8 P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: term changed from 0 to 1, leader changed from <none> to bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:44.907510 9626 raft_consensus.cc:1261] T eb9d11716a3341589cd827cad3b88fa5 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:44.907923 9363 raft_consensus.cc:1261] T eb9d11716a3341589cd827cad3b88fa5 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:44.908447 12677 consensus_queue.cc:1035] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:44.908648 12676 consensus_queue.cc:1035] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:44.911994 6737 catalog_manager.cc:5576] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087) does not have the latest schema for tablet eb9d11716a3341589cd827cad3b88fa5 (table rebalancer_dynamic_table [id=608ae4072a8041c5945a6202e671de59]). Expected version 1 got 0
I20240503 01:59:44.913100 6735 catalog_manager.cc:5576] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387) does not have the latest schema for tablet eb9d11716a3341589cd827cad3b88fa5 (table rebalancer_dynamic_table [id=608ae4072a8041c5945a6202e671de59]). Expected version 1 got 0
I20240503 01:59:44.915238 12717 tablet.cc:1722] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:44.916198 12717 tablet.cc:1620] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: MemRowSet was empty: no flush needed.
I20240503 01:59:44.917187 6737 catalog_manager.cc:5576] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387) does not have the latest schema for tablet eb9d11716a3341589cd827cad3b88fa5 (table rebalancer_dynamic_table [id=608ae4072a8041c5945a6202e671de59]). Expected version 1 got 0
I20240503 01:59:44.917512 12719 tablet.cc:1722] T eb9d11716a3341589cd827cad3b88fa5 P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:44.918735 12719 tablet.cc:1620] T eb9d11716a3341589cd827cad3b88fa5 P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 01:59:44.919023 6735 catalog_manager.cc:5576] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087) does not have the latest schema for tablet eb9d11716a3341589cd827cad3b88fa5 (table rebalancer_dynamic_table [id=608ae4072a8041c5945a6202e671de59]). Expected version 1 got 0
I20240503 01:59:44.921038 12718 tablet.cc:1722] T eb9d11716a3341589cd827cad3b88fa5 P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:44.921388 12718 tablet.cc:1620] T eb9d11716a3341589cd827cad3b88fa5 P b7062fbbc2b54fff9e99ec4ea5562c76: MemRowSet was empty: no flush needed.
I20240503 01:59:44.923794 6737 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=608ae4072a8041c5945a6202e671de59] alter complete (version 1)
I20240503 01:59:44.927091 7145 consensus_queue.cc:237] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:44.928506 9755 raft_consensus.cc:1261] T c6833c997d284063955d0b9b09ccb3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 2 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 01:59:44.928674 7014 raft_consensus.cc:1261] T c6833c997d284063955d0b9b09ccb3b7 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 01:59:44.928915 12676 consensus_queue.cc:1035] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:44.929136 12031 consensus_queue.cc:1035] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 01:59:44.931278 12677 raft_consensus.cc:2932] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:44.931787 9755 raft_consensus.cc:2932] T c6833c997d284063955d0b9b09ccb3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:44.933420 7014 raft_consensus.cc:2932] T c6833c997d284063955d0b9b09ccb3b7 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:44.933661 12484 raft_consensus.cc:482] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:44.933760 12484 raft_consensus.cc:504] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:44.934192 12484 leader_election.cc:288] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:44.934819 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb175f81d9e48229466503ebf7e3c09" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:44.934830 6736 catalog_manager.cc:5484] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index 4 to 6, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New cstate: current_term: 2 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:44.934988 9494 raft_consensus.cc:2454] T beb175f81d9e48229466503ebf7e3c09 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:44.935240 9626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb175f81d9e48229466503ebf7e3c09" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:44.935366 9626 raft_consensus.cc:2454] T beb175f81d9e48229466503ebf7e3c09 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:44.936636 9296 leader_election.cc:302] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:44.937273 12484 raft_consensus.cc:2781] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:44.937341 12484 raft_consensus.cc:482] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:44.937379 12484 raft_consensus.cc:3037] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:44.938544 12484 raft_consensus.cc:504] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:44.938859 7256 tablet_service.cc:1514] Processing DeleteTablet for tablet c6833c997d284063955d0b9b09ccb3b7 with delete_type TABLET_DATA_TOMBSTONED (TS 1d0e738c34af40fa9950c6f086619b3a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:44.938953 12484 leader_election.cc:288] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Requested vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:44.939411 9626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb175f81d9e48229466503ebf7e3c09" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:44.939558 9626 raft_consensus.cc:3037] T beb175f81d9e48229466503ebf7e3c09 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:44.940416 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb175f81d9e48229466503ebf7e3c09" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:44.940547 9494 raft_consensus.cc:3037] T beb175f81d9e48229466503ebf7e3c09 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:44.940640 9626 raft_consensus.cc:2454] T beb175f81d9e48229466503ebf7e3c09 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:44.941537 9494 raft_consensus.cc:2454] T beb175f81d9e48229466503ebf7e3c09 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:44.941874 9298 leader_election.cc:302] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:44.941990 12728 tablet_replica.cc:330] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:44.942066 12484 raft_consensus.cc:2781] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:44.942138 12728 raft_consensus.cc:2229] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:44.942257 12728 raft_consensus.cc:2258] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:44.942270 12484 raft_consensus.cc:686] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Running, Role: LEADER
I20240503 01:59:44.942412 12484 consensus_queue.cc:237] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:44.942845 12728 ts_tablet_manager.cc:1875] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:44.944743 6735 catalog_manager.cc:5484] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:44.946111 12728 ts_tablet_manager.cc:1888] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 01:59:44.946265 12728 log.cc:1196] T c6833c997d284063955d0b9b09ccb3b7 P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/c6833c997d284063955d0b9b09ccb3b7
I20240503 01:59:44.946823 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet c6833c997d284063955d0b9b09ccb3b7 (table rebalance_test_table_15 [id=76d0e687959345af9b8f30fe2c35d640]) successfully deleted
I20240503 01:59:45.025285 12677 raft_consensus.cc:482] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:45.025422 12677 raft_consensus.cc:504] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:45.026298 12677 leader_election.cc:288] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:45.026628 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43f13a33fe94033bce9debbae6990c2" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:45.026755 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43f13a33fe94033bce9debbae6990c2" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:45.026794 9494 raft_consensus.cc:2454] T c43f13a33fe94033bce9debbae6990c2 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:45.026875 9363 raft_consensus.cc:2454] T c43f13a33fe94033bce9debbae6990c2 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:45.027120 7078 leader_election.cc:302] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:45.027305 12677 raft_consensus.cc:2781] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:45.027352 12677 raft_consensus.cc:482] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:45.027401 12677 raft_consensus.cc:3037] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:45.028354 6736 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:45.028549 12677 raft_consensus.cc:504] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:45.028951 12677 leader_election.cc:288] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Requested vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:45.029536 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43f13a33fe94033bce9debbae6990c2" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:45.029529 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43f13a33fe94033bce9debbae6990c2" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:45.029651 9363 raft_consensus.cc:3037] T c43f13a33fe94033bce9debbae6990c2 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:45.029651 9494 raft_consensus.cc:3037] T c43f13a33fe94033bce9debbae6990c2 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:45.030696 9363 raft_consensus.cc:2454] T c43f13a33fe94033bce9debbae6990c2 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:45.030699 9494 raft_consensus.cc:2454] T c43f13a33fe94033bce9debbae6990c2 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:45.031018 7078 leader_election.cc:302] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:45.033025 12677 raft_consensus.cc:2781] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:45.033130 12677 raft_consensus.cc:686] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Becoming Leader. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Running, Role: LEADER
I20240503 01:59:45.033280 12677 consensus_queue.cc:237] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:45.036857 12717 tablet.cc:1722] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:45.037214 12717 tablet.cc:1620] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: MemRowSet was empty: no flush needed.
I20240503 01:59:45.038056 6741 catalog_manager.cc:5484] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:45.042991 12718 tablet.cc:1722] T eb9d11716a3341589cd827cad3b88fa5 P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:45.043386 12718 tablet.cc:1620] T eb9d11716a3341589cd827cad3b88fa5 P b7062fbbc2b54fff9e99ec4ea5562c76: MemRowSet was empty: no flush needed.
I20240503 01:59:45.043458 12719 tablet.cc:1722] T eb9d11716a3341589cd827cad3b88fa5 P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:45.043762 12719 tablet.cc:1620] T eb9d11716a3341589cd827cad3b88fa5 P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 01:59:45.050485 6732 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=608ae4072a8041c5945a6202e671de59] alter complete (version 2)
I20240503 01:59:45.109452 6861 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:45.124392 9342 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:45.125836 6993 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:45.128257 9737 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:45.130482 7256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:45.132272 9472 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:45.147874 7124 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:45.160570 6735 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:45.174942 12717 tablet.cc:1722] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:45.176795 12718 tablet.cc:1722] T eb9d11716a3341589cd827cad3b88fa5 P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:45.179694 12719 tablet.cc:1722] T eb9d11716a3341589cd827cad3b88fa5 P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:45.184708 6738 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=608ae4072a8041c5945a6202e671de59] alter complete (version 3)
I20240503 01:59:45.187893 9604 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:45.234715 7014 consensus_queue.cc:237] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:45.236101 7145 raft_consensus.cc:1261] T 5cb5072e91534671ad3b4dd85f93fa5b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:45.236068 9626 raft_consensus.cc:1261] T 5cb5072e91534671ad3b4dd85f93fa5b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:45.236531 12321 consensus_queue.cc:1035] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:45.236732 12809 consensus_queue.cc:1035] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:45.238497 12809 raft_consensus.cc:2932] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:45.239045 7145 raft_consensus.cc:2932] T 5cb5072e91534671ad3b4dd85f93fa5b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:45.239151 9626 raft_consensus.cc:2932] T 5cb5072e91534671ad3b4dd85f93fa5b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:45.241714 6735 catalog_manager.cc:5484] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:45.246567 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cb5072e91534671ad3b4dd85f93fa5b with delete_type TABLET_DATA_TOMBSTONED (TS 5dcc33cc621a4cbbae155f3230fb7c46 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:45.247231 12828 tablet_replica.cc:330] T 5cb5072e91534671ad3b4dd85f93fa5b P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:45.247471 12828 raft_consensus.cc:2229] T 5cb5072e91534671ad3b4dd85f93fa5b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:45.247678 12828 raft_consensus.cc:2258] T 5cb5072e91534671ad3b4dd85f93fa5b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:45.248756 12828 ts_tablet_manager.cc:1875] T 5cb5072e91534671ad3b4dd85f93fa5b P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:45.251088 12828 ts_tablet_manager.cc:1888] T 5cb5072e91534671ad3b4dd85f93fa5b P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 01:59:45.251174 12828 log.cc:1196] T 5cb5072e91534671ad3b4dd85f93fa5b P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/5cb5072e91534671ad3b4dd85f93fa5b
I20240503 01:59:45.251878 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 5cb5072e91534671ad3b4dd85f93fa5b (table rebalance_test_table_9 [id=f050ea79f4334dcda01dce5d3e804176]) successfully deleted
I20240503 01:59:45.288188 6732 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:45.288494 6732 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:45.290895 6732 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet eb9d11716a3341589cd827cad3b88fa5
I20240503 01:59:45.291546 9342 tablet_service.cc:1514] Processing DeleteTablet for tablet eb9d11716a3341589cd827cad3b88fa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:45 UTC) from {username='slave'} at 127.0.0.1:36040
I20240503 01:59:45.291715 7124 tablet_service.cc:1514] Processing DeleteTablet for tablet eb9d11716a3341589cd827cad3b88fa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:45 UTC) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:45.292062 12865 tablet_replica.cc:330] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:45.292294 9604 tablet_service.cc:1514] Processing DeleteTablet for tablet eb9d11716a3341589cd827cad3b88fa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:45 UTC) from {username='slave'} at 127.0.0.1:58562
I20240503 01:59:45.292348 12864 tablet_replica.cc:330] T eb9d11716a3341589cd827cad3b88fa5 P 12e8c683db054df3b75c6285b98a5ad8: stopping tablet replica
I20240503 01:59:45.292480 12864 raft_consensus.cc:2229] T eb9d11716a3341589cd827cad3b88fa5 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:45.292583 12866 tablet_replica.cc:330] T eb9d11716a3341589cd827cad3b88fa5 P b7062fbbc2b54fff9e99ec4ea5562c76: stopping tablet replica
I20240503 01:59:45.292657 12864 raft_consensus.cc:2258] T eb9d11716a3341589cd827cad3b88fa5 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:45.292567 12865 raft_consensus.cc:2229] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:45.292811 12866 raft_consensus.cc:2229] T eb9d11716a3341589cd827cad3b88fa5 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:45.292886 12865 raft_consensus.cc:2258] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:45.292976 12866 raft_consensus.cc:2258] T eb9d11716a3341589cd827cad3b88fa5 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:45.293846 12865 ts_tablet_manager.cc:1875] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:45.293905 12866 ts_tablet_manager.cc:1875] T eb9d11716a3341589cd827cad3b88fa5 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:45.293748 12864 ts_tablet_manager.cc:1875] T eb9d11716a3341589cd827cad3b88fa5 P 12e8c683db054df3b75c6285b98a5ad8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:45.296308 12866 ts_tablet_manager.cc:1888] T eb9d11716a3341589cd827cad3b88fa5 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:45.296437 12866 log.cc:1196] T eb9d11716a3341589cd827cad3b88fa5 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/eb9d11716a3341589cd827cad3b88fa5
I20240503 01:59:45.296770 12866 ts_tablet_manager.cc:1909] T eb9d11716a3341589cd827cad3b88fa5 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting consensus metadata
I20240503 01:59:45.297000 12865 ts_tablet_manager.cc:1888] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:45.297094 12865 log.cc:1196] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/eb9d11716a3341589cd827cad3b88fa5
I20240503 01:59:45.297433 12865 ts_tablet_manager.cc:1909] T eb9d11716a3341589cd827cad3b88fa5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting consensus metadata
I20240503 01:59:45.297580 6727 catalog_manager.cc:4830] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): tablet eb9d11716a3341589cd827cad3b88fa5 (table rebalancer_dynamic_table_new_name [id=608ae4072a8041c5945a6202e671de59]) successfully deleted
I20240503 01:59:45.298352 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet eb9d11716a3341589cd827cad3b88fa5 (table rebalancer_dynamic_table_new_name [id=608ae4072a8041c5945a6202e671de59]) successfully deleted
I20240503 01:59:45.302415 7014 consensus_queue.cc:237] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:45.304244 6881 raft_consensus.cc:1261] T 87652406a8e442e58fe636a85ab72158 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:45.304709 7143 raft_consensus.cc:1261] T 87652406a8e442e58fe636a85ab72158 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:45.304703 12321 consensus_queue.cc:1035] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:45.304955 6947 consensus_peers.cc:485] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87652406a8e442e58fe636a85ab72158. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:45.305199 12321 consensus_queue.cc:1035] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:45.307267 12321 raft_consensus.cc:2932] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:45.308369 12864 ts_tablet_manager.cc:1888] T eb9d11716a3341589cd827cad3b88fa5 P 12e8c683db054df3b75c6285b98a5ad8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:45.308667 12864 log.cc:1196] T eb9d11716a3341589cd827cad3b88fa5 P 12e8c683db054df3b75c6285b98a5ad8: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/eb9d11716a3341589cd827cad3b88fa5
I20240503 01:59:45.309231 12864 ts_tablet_manager.cc:1909] T eb9d11716a3341589cd827cad3b88fa5 P 12e8c683db054df3b75c6285b98a5ad8: Deleting consensus metadata
I20240503 01:59:45.309486 6881 raft_consensus.cc:2932] T 87652406a8e442e58fe636a85ab72158 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:45.312041 6732 catalog_manager.cc:5484] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:45.313941 6727 catalog_manager.cc:4830] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): tablet eb9d11716a3341589cd827cad3b88fa5 (table rebalancer_dynamic_table_new_name [id=608ae4072a8041c5945a6202e671de59]) successfully deleted
I20240503 01:59:45.314369 7143 raft_consensus.cc:2932] T 87652406a8e442e58fe636a85ab72158 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:45.339490 12070 consensus_queue.cc:1035] T 5ec2909a6e484f81aad7b39a374e1ef8 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:45.342506 7014 consensus_queue.cc:237] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:45.343994 6881 raft_consensus.cc:1261] T 85d3e89fac21402387aa812f754cf5bc P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:45.344657 12873 consensus_queue.cc:1035] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:45.344686 12707 consensus_queue.cc:1035] T 5ec2909a6e484f81aad7b39a374e1ef8 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 01:59:45.345144 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 85d3e89fac21402387aa812f754cf5bc. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:45.345265 7143 raft_consensus.cc:1261] T 85d3e89fac21402387aa812f754cf5bc P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:45.345867 12876 consensus_queue.cc:1035] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:45.352201 12809 raft_consensus.cc:2932] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:45.352613 7143 raft_consensus.cc:2932] T 85d3e89fac21402387aa812f754cf5bc P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:45.354359 6881 raft_consensus.cc:2932] T 85d3e89fac21402387aa812f754cf5bc P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:45.356828 6739 catalog_manager.cc:5484] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:45.384050 6881 consensus_queue.cc:237] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:45.386078 7143 raft_consensus.cc:1261] T 8ad236ea5ab347c69026293bef7c04a8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:45.386464 7275 raft_consensus.cc:1261] T 8ad236ea5ab347c69026293bef7c04a8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:45.386718 6816 consensus_peers.cc:485] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ad236ea5ab347c69026293bef7c04a8. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:45.387095 12504 consensus_queue.cc:1035] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:45.387723 12916 consensus_queue.cc:1035] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:45.390501 12504 raft_consensus.cc:2932] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:45.390973 7275 raft_consensus.cc:2932] T 8ad236ea5ab347c69026293bef7c04a8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:45.393019 6738 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:45.393304 7143 raft_consensus.cc:2932] T 8ad236ea5ab347c69026293bef7c04a8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:45.394395 6732 catalog_manager.cc:5484] T 8ad236ea5ab347c69026293bef7c04a8 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:45.402164 9472 tablet_service.cc:1467] Processing CreateTablet for tablet 1367d8a2e38844acaa6bf1c57324f6f9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e656a18417874f33aeff28650a93bbe7]), partition=
I20240503 01:59:45.402164 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 1367d8a2e38844acaa6bf1c57324f6f9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e656a18417874f33aeff28650a93bbe7]), partition=
I20240503 01:59:45.402459 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1367d8a2e38844acaa6bf1c57324f6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:45.402482 9472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1367d8a2e38844acaa6bf1c57324f6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:45.404937 12926 tablet_bootstrap.cc:492] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:45.405369 9604 tablet_service.cc:1467] Processing CreateTablet for tablet 1367d8a2e38844acaa6bf1c57324f6f9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e656a18417874f33aeff28650a93bbe7]), partition=
I20240503 01:59:45.405674 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1367d8a2e38844acaa6bf1c57324f6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:45.406729 12926 tablet_bootstrap.cc:654] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:45.406729 12925 tablet_bootstrap.cc:492] T 1367d8a2e38844acaa6bf1c57324f6f9 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:45.408363 12927 tablet_bootstrap.cc:492] T 1367d8a2e38844acaa6bf1c57324f6f9 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:45.409135 12925 tablet_bootstrap.cc:654] T 1367d8a2e38844acaa6bf1c57324f6f9 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:45.409369 12927 tablet_bootstrap.cc:654] T 1367d8a2e38844acaa6bf1c57324f6f9 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:45.410708 12926 tablet_bootstrap.cc:492] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:45.410820 12925 tablet_bootstrap.cc:492] T 1367d8a2e38844acaa6bf1c57324f6f9 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:45.410868 12926 ts_tablet_manager.cc:1367] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 01:59:45.410879 12927 tablet_bootstrap.cc:492] T 1367d8a2e38844acaa6bf1c57324f6f9 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:45.410969 12925 ts_tablet_manager.cc:1367] T 1367d8a2e38844acaa6bf1c57324f6f9 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 01:59:45.411263 12927 ts_tablet_manager.cc:1367] T 1367d8a2e38844acaa6bf1c57324f6f9 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:45.411470 12926 raft_consensus.cc:348] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:45.411602 12926 raft_consensus.cc:374] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:45.411644 12926 raft_consensus.cc:729] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:45.411582 12925 raft_consensus.cc:348] T 1367d8a2e38844acaa6bf1c57324f6f9 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:45.411706 12925 raft_consensus.cc:374] T 1367d8a2e38844acaa6bf1c57324f6f9 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:45.411823 12925 raft_consensus.cc:729] T 1367d8a2e38844acaa6bf1c57324f6f9 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:45.411818 12926 consensus_queue.cc:260] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:45.411998 12925 consensus_queue.cc:260] T 1367d8a2e38844acaa6bf1c57324f6f9 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:45.411993 12927 raft_consensus.cc:348] T 1367d8a2e38844acaa6bf1c57324f6f9 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:45.412086 12927 raft_consensus.cc:374] T 1367d8a2e38844acaa6bf1c57324f6f9 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:45.412154 12927 raft_consensus.cc:729] T 1367d8a2e38844acaa6bf1c57324f6f9 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:45.412374 12927 consensus_queue.cc:260] T 1367d8a2e38844acaa6bf1c57324f6f9 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:45.412459 12925 ts_tablet_manager.cc:1398] T 1367d8a2e38844acaa6bf1c57324f6f9 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:45.412786 12927 ts_tablet_manager.cc:1398] T 1367d8a2e38844acaa6bf1c57324f6f9 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:45.413420 12926 ts_tablet_manager.cc:1398] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:45.427939 12733 consensus_queue.cc:1035] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:45.430475 7143 consensus_queue.cc:237] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:45.431936 12484 consensus_queue.cc:1035] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:45.432062 7275 raft_consensus.cc:1261] T dfdcdd54606e4320a5d0f506eb6dabf8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:45.432129 7080 consensus_peers.cc:485] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dfdcdd54606e4320a5d0f506eb6dabf8. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:45.432547 12676 consensus_queue.cc:1035] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:45.433692 6881 raft_consensus.cc:1261] T dfdcdd54606e4320a5d0f506eb6dabf8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:45.434257 12676 consensus_queue.cc:1035] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:45.436290 12031 raft_consensus.cc:2932] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:45.436893 6881 raft_consensus.cc:2932] T dfdcdd54606e4320a5d0f506eb6dabf8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:45.438509 7275 raft_consensus.cc:2932] T dfdcdd54606e4320a5d0f506eb6dabf8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:45.439941 6739 catalog_manager.cc:5484] T dfdcdd54606e4320a5d0f506eb6dabf8 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:45.464040 7275 consensus_queue.cc:237] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:45.465452 9494 raft_consensus.cc:1261] T 9cc3435afe5144f9a52a286ac86d234f P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:45.465487 9363 raft_consensus.cc:1261] T 9cc3435afe5144f9a52a286ac86d234f P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:45.465902 12963 consensus_queue.cc:1035] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:45.466080 12964 consensus_queue.cc:1035] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 01:59:45.466291 7209 consensus_peers.cc:485] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a -> Peer 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): Couldn't send request to peer 63331f455be049c1bdd6c23c59ac1bef. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9cc3435afe5144f9a52a286ac86d234f. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:45.467175 12707 raft_consensus.cc:482] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:45.467267 12707 raft_consensus.cc:504] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:45.467581 12707 leader_election.cc:288] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:45.468011 9626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1367d8a2e38844acaa6bf1c57324f6f9" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:45.468118 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1367d8a2e38844acaa6bf1c57324f6f9" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:45.468163 9626 raft_consensus.cc:2454] T 1367d8a2e38844acaa6bf1c57324f6f9 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:45.468248 9363 raft_consensus.cc:2454] T 1367d8a2e38844acaa6bf1c57324f6f9 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:45.468456 9429 leader_election.cc:302] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:45.468557 12964 raft_consensus.cc:2932] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:45.468756 12707 raft_consensus.cc:2781] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:45.468815 12707 raft_consensus.cc:482] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:45.468851 12707 raft_consensus.cc:3037] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:45.469010 9363 raft_consensus.cc:2932] T 9cc3435afe5144f9a52a286ac86d234f P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:45.469769 12707 raft_consensus.cc:504] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:45.470161 12707 leader_election.cc:288] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Requested vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:45.470971 9494 raft_consensus.cc:2932] T 9cc3435afe5144f9a52a286ac86d234f P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:45.471159 9626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1367d8a2e38844acaa6bf1c57324f6f9" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:45.471278 9626 raft_consensus.cc:3037] T 1367d8a2e38844acaa6bf1c57324f6f9 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:45.473027 6735 catalog_manager.cc:5484] T 9cc3435afe5144f9a52a286ac86d234f P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:45.473600 9626 raft_consensus.cc:2454] T 1367d8a2e38844acaa6bf1c57324f6f9 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:45.473953 9429 leader_election.cc:302] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:45.474205 12070 raft_consensus.cc:2781] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:45.474280 12070 raft_consensus.cc:686] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Becoming Leader. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Running, Role: LEADER
I20240503 01:59:45.474404 12070 consensus_queue.cc:237] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:45.475878 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1367d8a2e38844acaa6bf1c57324f6f9" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:45.476006 9363 raft_consensus.cc:3037] T 1367d8a2e38844acaa6bf1c57324f6f9 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:45.477559 12946 consensus_queue.cc:1035] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:45.477756 9363 raft_consensus.cc:2454] T 1367d8a2e38844acaa6bf1c57324f6f9 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:45.478484 6739 catalog_manager.cc:5484] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: term changed from 0 to 1, leader changed from <none> to bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:45.481523 12946 consensus_queue.cc:1035] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:45.503412 7014 consensus_queue.cc:237] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:45.504947 7143 raft_consensus.cc:1261] T 4be54233e31248cd8deafa6b7845a3f8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:45.505160 6948 consensus_peers.cc:485] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef -> Peer db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): Couldn't send request to peer db1a0000aaa346f79c0aa4b6c5105191. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4be54233e31248cd8deafa6b7845a3f8. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:45.505293 7275 raft_consensus.cc:1261] T 4be54233e31248cd8deafa6b7845a3f8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:45.506021 12873 consensus_queue.cc:1035] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:45.506234 12809 consensus_queue.cc:1035] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:45.510293 12809 raft_consensus.cc:2932] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:45.510707 7275 raft_consensus.cc:2932] T 4be54233e31248cd8deafa6b7845a3f8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:45.513636 6732 catalog_manager.cc:5484] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:45.516942 7143 raft_consensus.cc:2932] T 4be54233e31248cd8deafa6b7845a3f8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:45.540982 9363 consensus_queue.cc:237] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
W20240503 01:59:45.543030 9299 consensus_peers.cc:485] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 -> Peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): Couldn't send request to peer 5dcc33cc621a4cbbae155f3230fb7c46. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b6f98b01ead4cea8f362dc283caa901. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:45.543020 9494 raft_consensus.cc:1261] T 1b6f98b01ead4cea8f362dc283caa901 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:45.543830 9755 raft_consensus.cc:1261] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:45.543516 13005 consensus_queue.cc:1035] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:45.544028 6739 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:45.544207 12484 consensus_queue.cc:1035] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:45.552793 6993 tablet_service.cc:1467] Processing CreateTablet for tablet b5329f65c13c47d485cee81cf9cf4f1e (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=cad604d7e464473fa089f248c9623507]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:45.553143 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5329f65c13c47d485cee81cf9cf4f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:45.553190 9604 tablet_service.cc:1467] Processing CreateTablet for tablet b5329f65c13c47d485cee81cf9cf4f1e (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=cad604d7e464473fa089f248c9623507]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:45.553478 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5329f65c13c47d485cee81cf9cf4f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:45.554803 12484 raft_consensus.cc:2932] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:45.555974 12927 tablet_bootstrap.cc:492] T b5329f65c13c47d485cee81cf9cf4f1e P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:45.556191 9737 tablet_service.cc:1467] Processing CreateTablet for tablet b5329f65c13c47d485cee81cf9cf4f1e (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=cad604d7e464473fa089f248c9623507]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:45.556483 9737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5329f65c13c47d485cee81cf9cf4f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:45.557015 12927 tablet_bootstrap.cc:654] T b5329f65c13c47d485cee81cf9cf4f1e P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:45.557535 9494 raft_consensus.cc:2932] T 1b6f98b01ead4cea8f362dc283caa901 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:45.558131 12927 tablet_bootstrap.cc:492] T b5329f65c13c47d485cee81cf9cf4f1e P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:45.558230 12927 ts_tablet_manager.cc:1367] T b5329f65c13c47d485cee81cf9cf4f1e P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:45.558301 9755 raft_consensus.cc:2932] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:45.558884 12927 raft_consensus.cc:348] T b5329f65c13c47d485cee81cf9cf4f1e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:45.558990 12927 raft_consensus.cc:374] T b5329f65c13c47d485cee81cf9cf4f1e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:45.559026 12927 raft_consensus.cc:729] T b5329f65c13c47d485cee81cf9cf4f1e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:45.559154 12927 consensus_queue.cc:260] T b5329f65c13c47d485cee81cf9cf4f1e P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:45.559465 12927 ts_tablet_manager.cc:1398] T b5329f65c13c47d485cee81cf9cf4f1e P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:45.560065 7256 tablet_service.cc:1467] Processing CreateTablet for tablet 719808dd1cb141a8b0538156bddd2761 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=cad604d7e464473fa089f248c9623507]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:45.560384 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 719808dd1cb141a8b0538156bddd2761. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:45.560781 7124 tablet_service.cc:1467] Processing CreateTablet for tablet 7672f42acd9446ef85eced07729c197b (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=cad604d7e464473fa089f248c9623507]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:45.561033 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7672f42acd9446ef85eced07729c197b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:45.561199 9736 tablet_service.cc:1467] Processing CreateTablet for tablet 719808dd1cb141a8b0538156bddd2761 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=cad604d7e464473fa089f248c9623507]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:45.561450 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 719808dd1cb141a8b0538156bddd2761. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:45.563638 9735 tablet_service.cc:1467] Processing CreateTablet for tablet 7672f42acd9446ef85eced07729c197b (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=cad604d7e464473fa089f248c9623507]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:45.563890 9735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7672f42acd9446ef85eced07729c197b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:45.564826 6732 catalog_manager.cc:5484] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:45.566829 13012 tablet_bootstrap.cc:492] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:45.567919 13012 tablet_bootstrap.cc:654] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:45.569185 13012 tablet_bootstrap.cc:492] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:45.569272 13012 ts_tablet_manager.cc:1367] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:45.569429 9472 tablet_service.cc:1467] Processing CreateTablet for tablet 719808dd1cb141a8b0538156bddd2761 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=cad604d7e464473fa089f248c9623507]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:45.569723 9472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 719808dd1cb141a8b0538156bddd2761. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:45.569723 13012 raft_consensus.cc:348] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:45.569828 13012 raft_consensus.cc:374] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:45.569857 13012 raft_consensus.cc:729] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:45.569964 13012 consensus_queue.cc:260] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:45.570268 13012 ts_tablet_manager.cc:1398] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:45.570083 6993 tablet_service.cc:1467] Processing CreateTablet for tablet 7672f42acd9446ef85eced07729c197b (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=cad604d7e464473fa089f248c9623507]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:45.571321 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7672f42acd9446ef85eced07729c197b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:45.572285 12926 tablet_bootstrap.cc:492] T 719808dd1cb141a8b0538156bddd2761 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:45.572295 11786 raft_consensus.cc:482] T b5329f65c13c47d485cee81cf9cf4f1e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:45.572377 11786 raft_consensus.cc:504] T b5329f65c13c47d485cee81cf9cf4f1e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:45.572772 11786 leader_election.cc:288] T b5329f65c13c47d485cee81cf9cf4f1e P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:45.573302 12926 tablet_bootstrap.cc:654] T 719808dd1cb141a8b0538156bddd2761 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:45.573339 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5329f65c13c47d485cee81cf9cf4f1e" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
W20240503 01:59:45.573750 9560 leader_election.cc:341] T b5329f65c13c47d485cee81cf9cf4f1e P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): Illegal state: must be running to vote when last-logged opid is not known
I20240503 01:59:45.574340 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5329f65c13c47d485cee81cf9cf4f1e" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
W20240503 01:59:45.574751 9559 leader_election.cc:341] T b5329f65c13c47d485cee81cf9cf4f1e P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): Illegal state: must be running to vote when last-logged opid is not known
I20240503 01:59:45.574827 9559 leader_election.cc:302] T b5329f65c13c47d485cee81cf9cf4f1e P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76; no voters: 63331f455be049c1bdd6c23c59ac1bef, db1a0000aaa346f79c0aa4b6c5105191
I20240503 01:59:45.575035 11786 raft_consensus.cc:2726] T b5329f65c13c47d485cee81cf9cf4f1e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 01:59:45.577925 12926 tablet_bootstrap.cc:492] T 719808dd1cb141a8b0538156bddd2761 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:45.578063 12926 ts_tablet_manager.cc:1367] T 719808dd1cb141a8b0538156bddd2761 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 01:59:45.578598 12926 raft_consensus.cc:348] T 719808dd1cb141a8b0538156bddd2761 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:45.578724 12926 raft_consensus.cc:374] T 719808dd1cb141a8b0538156bddd2761 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:45.578763 12926 raft_consensus.cc:729] T 719808dd1cb141a8b0538156bddd2761 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:45.578886 12926 consensus_queue.cc:260] T 719808dd1cb141a8b0538156bddd2761 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:45.578969 13020 tablet_bootstrap.cc:492] T 719808dd1cb141a8b0538156bddd2761 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:45.579180 12926 ts_tablet_manager.cc:1398] T 719808dd1cb141a8b0538156bddd2761 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:45.580088 13020 tablet_bootstrap.cc:654] T 719808dd1cb141a8b0538156bddd2761 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:45.582883 13008 tablet_bootstrap.cc:492] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:45.583837 13008 tablet_bootstrap.cc:654] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:45.584602 13013 tablet_bootstrap.cc:492] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:45.585973 13013 tablet_bootstrap.cc:654] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:45.587280 13013 tablet_bootstrap.cc:492] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:45.587443 13013 ts_tablet_manager.cc:1367] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 01:59:45.587636 13020 tablet_bootstrap.cc:492] T 719808dd1cb141a8b0538156bddd2761 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:45.587734 13020 ts_tablet_manager.cc:1367] T 719808dd1cb141a8b0538156bddd2761 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 01:59:45.588189 13013 raft_consensus.cc:348] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:45.588338 13013 raft_consensus.cc:374] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:45.588272 13020 raft_consensus.cc:348] T 719808dd1cb141a8b0538156bddd2761 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:45.588387 13020 raft_consensus.cc:374] T 719808dd1cb141a8b0538156bddd2761 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:45.588397 13013 raft_consensus.cc:729] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:45.588428 13020 raft_consensus.cc:729] T 719808dd1cb141a8b0538156bddd2761 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:45.588541 13020 consensus_queue.cc:260] T 719808dd1cb141a8b0538156bddd2761 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:45.588543 13013 consensus_queue.cc:260] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:45.588837 13013 ts_tablet_manager.cc:1398] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:45.588850 13020 ts_tablet_manager.cc:1398] T 719808dd1cb141a8b0538156bddd2761 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:45.588989 13013 tablet_bootstrap.cc:492] T 7672f42acd9446ef85eced07729c197b P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:45.589941 13013 tablet_bootstrap.cc:654] T 7672f42acd9446ef85eced07729c197b P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:45.590437 13008 tablet_bootstrap.cc:492] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:45.590560 13008 ts_tablet_manager.cc:1367] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 01:59:45.591084 13008 raft_consensus.cc:348] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:45.591210 13008 raft_consensus.cc:374] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:45.591248 13008 raft_consensus.cc:729] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:45.591392 13008 consensus_queue.cc:260] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:45.591681 13008 ts_tablet_manager.cc:1398] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:45.591842 13008 tablet_bootstrap.cc:492] T 7672f42acd9446ef85eced07729c197b P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:45.592106 13013 tablet_bootstrap.cc:492] T 7672f42acd9446ef85eced07729c197b P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:45.592180 13013 ts_tablet_manager.cc:1367] T 7672f42acd9446ef85eced07729c197b P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:59:45.592646 13013 raft_consensus.cc:348] T 7672f42acd9446ef85eced07729c197b P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:45.592801 13013 raft_consensus.cc:374] T 7672f42acd9446ef85eced07729c197b P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:45.592814 13008 tablet_bootstrap.cc:654] T 7672f42acd9446ef85eced07729c197b P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:45.592839 13013 raft_consensus.cc:729] T 7672f42acd9446ef85eced07729c197b P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:45.592976 13013 consensus_queue.cc:260] T 7672f42acd9446ef85eced07729c197b P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:45.593257 13013 ts_tablet_manager.cc:1398] T 7672f42acd9446ef85eced07729c197b P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:45.593446 13013 tablet_bootstrap.cc:492] T b5329f65c13c47d485cee81cf9cf4f1e P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:45.594592 13013 tablet_bootstrap.cc:654] T b5329f65c13c47d485cee81cf9cf4f1e P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:45.596196 6736 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:45.598513 13008 tablet_bootstrap.cc:492] T 7672f42acd9446ef85eced07729c197b P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:45.598712 13008 ts_tablet_manager.cc:1367] T 7672f42acd9446ef85eced07729c197b P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:59:45.599233 13008 raft_consensus.cc:348] T 7672f42acd9446ef85eced07729c197b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:45.599390 13008 raft_consensus.cc:374] T 7672f42acd9446ef85eced07729c197b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:45.599452 13008 raft_consensus.cc:729] T 7672f42acd9446ef85eced07729c197b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:45.599649 13008 consensus_queue.cc:260] T 7672f42acd9446ef85eced07729c197b P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:45.599931 13013 tablet_bootstrap.cc:492] T b5329f65c13c47d485cee81cf9cf4f1e P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:45.600046 13013 ts_tablet_manager.cc:1367] T b5329f65c13c47d485cee81cf9cf4f1e P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 01:59:45.600762 13008 ts_tablet_manager.cc:1398] T 7672f42acd9446ef85eced07729c197b P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:45.601580 13013 raft_consensus.cc:348] T b5329f65c13c47d485cee81cf9cf4f1e P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:45.601696 13013 raft_consensus.cc:374] T b5329f65c13c47d485cee81cf9cf4f1e P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:45.601729 13013 raft_consensus.cc:729] T b5329f65c13c47d485cee81cf9cf4f1e P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:45.601821 9626 raft_consensus.cc:1261] T 1367d8a2e38844acaa6bf1c57324f6f9 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:45.601878 9363 raft_consensus.cc:1261] T 1367d8a2e38844acaa6bf1c57324f6f9 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:45.601872 13013 consensus_queue.cc:260] T b5329f65c13c47d485cee81cf9cf4f1e P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:45.602279 12070 consensus_queue.cc:1035] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:45.602603 12070 consensus_queue.cc:1035] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:45.602691 13013 ts_tablet_manager.cc:1398] T b5329f65c13c47d485cee81cf9cf4f1e P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 01:59:45.606071 6736 catalog_manager.cc:5576] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387) does not have the latest schema for tablet 1367d8a2e38844acaa6bf1c57324f6f9 (table rebalancer_dynamic_table [id=e656a18417874f33aeff28650a93bbe7]). Expected version 1 got 0
I20240503 01:59:45.607452 6732 catalog_manager.cc:5576] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087) does not have the latest schema for tablet 1367d8a2e38844acaa6bf1c57324f6f9 (table rebalancer_dynamic_table [id=e656a18417874f33aeff28650a93bbe7]). Expected version 1 got 0
I20240503 01:59:45.612040 13035 tablet.cc:1722] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:45.612473 13035 tablet.cc:1620] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:45.613190 12718 tablet.cc:1722] T 1367d8a2e38844acaa6bf1c57324f6f9 P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:45.613507 12718 tablet.cc:1620] T 1367d8a2e38844acaa6bf1c57324f6f9 P b7062fbbc2b54fff9e99ec4ea5562c76: MemRowSet was empty: no flush needed.
I20240503 01:59:45.613793 12719 tablet.cc:1722] T 1367d8a2e38844acaa6bf1c57324f6f9 P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:45.614054 12719 tablet.cc:1620] T 1367d8a2e38844acaa6bf1c57324f6f9 P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 01:59:45.615823 6739 catalog_manager.cc:5576] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387) does not have the latest schema for tablet 1367d8a2e38844acaa6bf1c57324f6f9 (table rebalancer_dynamic_table [id=e656a18417874f33aeff28650a93bbe7]). Expected version 1 got 0
I20240503 01:59:45.618983 6732 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=e656a18417874f33aeff28650a93bbe7] alter complete (version 1)
I20240503 01:59:45.624372 12462 raft_consensus.cc:482] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:45.624691 12462 raft_consensus.cc:504] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:45.625068 7143 consensus_queue.cc:237] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:45.625314 12462 leader_election.cc:288] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:45.625797 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "719808dd1cb141a8b0538156bddd2761" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:45.625875 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "719808dd1cb141a8b0538156bddd2761" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:45.625994 9494 raft_consensus.cc:2454] T 719808dd1cb141a8b0538156bddd2761 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 01:59:45.626050 7275 raft_consensus.cc:2454] T 719808dd1cb141a8b0538156bddd2761 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 01:59:45.626364 9690 leader_election.cc:302] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:45.626778 12462 raft_consensus.cc:2781] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:45.626843 12462 raft_consensus.cc:482] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:45.626881 12462 raft_consensus.cc:3037] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:45.627494 9755 raft_consensus.cc:1261] T 84812ece4f27484eb008bfbebdbfee84 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 01:59:45.627804 7079 consensus_peers.cc:485] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): Couldn't send request to peer 1d0e738c34af40fa9950c6f086619b3a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84812ece4f27484eb008bfbebdbfee84. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:45.628110 12462 raft_consensus.cc:504] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:45.628266 9363 raft_consensus.cc:1261] T 84812ece4f27484eb008bfbebdbfee84 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:45.628499 12462 leader_election.cc:288] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:45.628646 12676 consensus_queue.cc:1035] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:45.628949 12031 consensus_queue.cc:1035] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:45.628973 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "719808dd1cb141a8b0538156bddd2761" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:45.628964 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "719808dd1cb141a8b0538156bddd2761" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:45.629101 7275 raft_consensus.cc:3037] T 719808dd1cb141a8b0538156bddd2761 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:45.629108 9494 raft_consensus.cc:3037] T 719808dd1cb141a8b0538156bddd2761 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:45.630183 7275 raft_consensus.cc:2454] T 719808dd1cb141a8b0538156bddd2761 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:45.630193 9494 raft_consensus.cc:2454] T 719808dd1cb141a8b0538156bddd2761 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:45.630439 12031 raft_consensus.cc:482] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:45.630550 12031 raft_consensus.cc:504] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:45.630595 9691 leader_election.cc:302] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:45.630779 12462 raft_consensus.cc:2781] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:45.630950 12462 raft_consensus.cc:686] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Becoming Leader. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Running, Role: LEADER
I20240503 01:59:45.630990 12031 leader_election.cc:288] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:45.631325 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7672f42acd9446ef85eced07729c197b" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:45.631487 9757 raft_consensus.cc:2454] T 7672f42acd9446ef85eced07729c197b P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:45.631552 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7672f42acd9446ef85eced07729c197b" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:45.631700 7014 raft_consensus.cc:2454] T 7672f42acd9446ef85eced07729c197b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:45.631775 7079 leader_election.cc:302] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:45.631942 12031 raft_consensus.cc:2781] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:45.631994 12031 raft_consensus.cc:482] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:45.632035 12031 raft_consensus.cc:3037] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:45.632970 12031 raft_consensus.cc:504] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:45.633306 12031 leader_election.cc:288] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Requested vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:45.631316 12462 consensus_queue.cc:237] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:45.633608 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7672f42acd9446ef85eced07729c197b" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:45.633692 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7672f42acd9446ef85eced07729c197b" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:45.633755 9755 raft_consensus.cc:3037] T 7672f42acd9446ef85eced07729c197b P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:45.633812 7014 raft_consensus.cc:3037] T 7672f42acd9446ef85eced07729c197b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:45.634778 9755 raft_consensus.cc:2454] T 7672f42acd9446ef85eced07729c197b P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:45.634778 7014 raft_consensus.cc:2454] T 7672f42acd9446ef85eced07729c197b P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:45.635149 7078 leader_election.cc:302] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:45.635310 12031 raft_consensus.cc:2932] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:45.635903 6736 catalog_manager.cc:5484] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: term changed from 0 to 1, leader changed from <none> to db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:45.636615 12873 raft_consensus.cc:482] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:45.636719 12873 raft_consensus.cc:504] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:45.637080 12676 raft_consensus.cc:2781] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:45.637148 12873 leader_election.cc:288] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:45.637176 12676 raft_consensus.cc:686] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Becoming Leader. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Running, Role: LEADER
I20240503 01:59:45.637311 12676 consensus_queue.cc:237] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:45.637799 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5329f65c13c47d485cee81cf9cf4f1e" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:45.637936 9625 raft_consensus.cc:2454] T b5329f65c13c47d485cee81cf9cf4f1e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:45.638403 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5329f65c13c47d485cee81cf9cf4f1e" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:45.638569 9755 raft_consensus.cc:2454] T b5329f65c13c47d485cee81cf9cf4f1e P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:45.638908 6948 leader_election.cc:302] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:45.638864 9757 raft_consensus.cc:2932] T 84812ece4f27484eb008bfbebdbfee84 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:45.639250 12321 raft_consensus.cc:2781] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:45.639297 12321 raft_consensus.cc:482] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:45.639331 12321 raft_consensus.cc:3037] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:45.640352 12321 raft_consensus.cc:504] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:45.640759 12321 leader_election.cc:288] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:45.641010 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5329f65c13c47d485cee81cf9cf4f1e" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:45.641129 9625 raft_consensus.cc:3037] T b5329f65c13c47d485cee81cf9cf4f1e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:45.641106 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5329f65c13c47d485cee81cf9cf4f1e" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:45.641217 9757 raft_consensus.cc:3037] T b5329f65c13c47d485cee81cf9cf4f1e P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:45.642151 9625 raft_consensus.cc:2454] T b5329f65c13c47d485cee81cf9cf4f1e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:45.642179 9757 raft_consensus.cc:2454] T b5329f65c13c47d485cee81cf9cf4f1e P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:45.642467 6948 leader_election.cc:302] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:45.643543 12873 raft_consensus.cc:2781] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:45.643638 12873 raft_consensus.cc:686] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:45.643775 12873 consensus_queue.cc:237] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:45.644428 9363 raft_consensus.cc:2932] T 84812ece4f27484eb008bfbebdbfee84 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:45.645700 6739 catalog_manager.cc:5484] T 84812ece4f27484eb008bfbebdbfee84 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:45.647117 6732 catalog_manager.cc:5484] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:45.650774 6736 catalog_manager.cc:5484] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:45.668934 9757 consensus_queue.cc:237] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:45.670914 9363 raft_consensus.cc:1261] T fad1113f16ba45009ae409f57eac4550 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 01:59:45.671144 9692 consensus_peers.cc:485] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): Couldn't send request to peer c688f3df56ac4b6d9b0c30fa4cc8ca2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fad1113f16ba45009ae409f57eac4550. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:45.671139 9625 raft_consensus.cc:1261] T fad1113f16ba45009ae409f57eac4550 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:45.671358 12462 consensus_queue.cc:1035] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:45.671590 13064 consensus_queue.cc:1035] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:45.673633 13064 raft_consensus.cc:2932] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:45.674180 9625 raft_consensus.cc:2932] T fad1113f16ba45009ae409f57eac4550 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:45.676199 9363 raft_consensus.cc:2932] T fad1113f16ba45009ae409f57eac4550 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:45.677073 6736 catalog_manager.cc:5484] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:45.698371 6881 consensus_queue.cc:237] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:45.700067 9757 raft_consensus.cc:1261] T 4e686819aed5407bacc13b3e3a0d4566 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:45.700264 9494 raft_consensus.cc:1261] T 4e686819aed5407bacc13b3e3a0d4566 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:45.700628 12916 consensus_queue.cc:1035] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:45.700809 12504 consensus_queue.cc:1035] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 01:59:45.700970 6816 consensus_peers.cc:485] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): Couldn't send request to peer 63331f455be049c1bdd6c23c59ac1bef. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e686819aed5407bacc13b3e3a0d4566. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:45.702841 12916 raft_consensus.cc:2932] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:45.703193 9757 raft_consensus.cc:2932] T 4e686819aed5407bacc13b3e3a0d4566 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:45.704690 9494 raft_consensus.cc:2932] T 4e686819aed5407bacc13b3e3a0d4566 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:45.707199 6738 catalog_manager.cc:5484] T 4e686819aed5407bacc13b3e3a0d4566 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:45.724920 6736 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:45.729326 7143 consensus_queue.cc:237] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:45.730751 13035 tablet.cc:1722] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
W20240503 01:59:45.731074 7079 consensus_peers.cc:485] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): Couldn't send request to peer db1a0000aaa346f79c0aa4b6c5105191. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79efb54e1f8143be868e9ef37f5f925d. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:45.731099 13035 tablet.cc:1620] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:45.731328 7275 raft_consensus.cc:1261] T 79efb54e1f8143be868e9ef37f5f925d P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:45.731549 12718 tablet.cc:1722] T 1367d8a2e38844acaa6bf1c57324f6f9 P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:45.731832 12718 tablet.cc:1620] T 1367d8a2e38844acaa6bf1c57324f6f9 P b7062fbbc2b54fff9e99ec4ea5562c76: MemRowSet was empty: no flush needed.
I20240503 01:59:45.732218 6881 raft_consensus.cc:1261] T 79efb54e1f8143be868e9ef37f5f925d P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:45.732386 12719 tablet.cc:1722] T 1367d8a2e38844acaa6bf1c57324f6f9 P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:45.732549 12031 consensus_queue.cc:1035] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:45.732707 12719 tablet.cc:1620] T 1367d8a2e38844acaa6bf1c57324f6f9 P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 01:59:45.733315 12031 consensus_queue.cc:1035] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:45.737576 6732 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=e656a18417874f33aeff28650a93bbe7] alter complete (version 2)
I20240503 01:59:45.738948 12946 raft_consensus.cc:2932] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:45.739542 7274 raft_consensus.cc:2932] T 79efb54e1f8143be868e9ef37f5f925d P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:45.741353 6881 raft_consensus.cc:2932] T 79efb54e1f8143be868e9ef37f5f925d P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:45.743149 6736 catalog_manager.cc:5484] T 79efb54e1f8143be868e9ef37f5f925d P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:45.846069 6733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:45.847978 6993 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:45.850150 7256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:45.851397 9342 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:45.856662 13035 tablet.cc:1722] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:45.858055 12719 tablet.cc:1722] T 1367d8a2e38844acaa6bf1c57324f6f9 P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:45.859189 12718 tablet.cc:1722] T 1367d8a2e38844acaa6bf1c57324f6f9 P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:45.864123 9604 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:45.864437 7124 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:45.865041 6733 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=e656a18417874f33aeff28650a93bbe7] alter complete (version 3)
I20240503 01:59:45.865276 6861 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:45.865437 9737 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:45.866189 9472 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:45.882715 13160 ts_tablet_manager.cc:902] T dfdcdd54606e4320a5d0f506eb6dabf8 P b7062fbbc2b54fff9e99ec4ea5562c76: Initiating tablet copy from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:45.883316 13160 tablet_copy_client.cc:308] T dfdcdd54606e4320a5d0f506eb6dabf8 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41979
I20240503 01:59:45.883822 7165 tablet_copy_service.cc:140] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Received BeginTabletCopySession request for tablet dfdcdd54606e4320a5d0f506eb6dabf8 from peer b7062fbbc2b54fff9e99ec4ea5562c76 ({username='slave'} at 127.0.99.135:42797)
I20240503 01:59:45.883929 7165 tablet_copy_service.cc:161] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Beginning new tablet copy session on tablet dfdcdd54606e4320a5d0f506eb6dabf8 from peer b7062fbbc2b54fff9e99ec4ea5562c76 at {username='slave'} at 127.0.99.135:42797: session id = b7062fbbc2b54fff9e99ec4ea5562c76-dfdcdd54606e4320a5d0f506eb6dabf8
I20240503 01:59:45.884780 7165 tablet_copy_source_session.cc:205] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:45.885928 13160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfdcdd54606e4320a5d0f506eb6dabf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:45.888513 13160 tablet_copy_client.cc:783] T dfdcdd54606e4320a5d0f506eb6dabf8 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:45.888779 13160 tablet_copy_client.cc:647] T dfdcdd54606e4320a5d0f506eb6dabf8 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:45.904038 13160 tablet_copy_client.cc:523] T dfdcdd54606e4320a5d0f506eb6dabf8 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:45.905870 13160 tablet_bootstrap.cc:492] T dfdcdd54606e4320a5d0f506eb6dabf8 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:45.909507 13165 ts_tablet_manager.cc:902] T 87652406a8e442e58fe636a85ab72158 P bb5364d730cb4489a0bfc6c270b7450a: Initiating tablet copy from peer 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:45.927971 13158 ts_tablet_manager.cc:902] T 85d3e89fac21402387aa812f754cf5bc P 12e8c683db054df3b75c6285b98a5ad8: Initiating tablet copy from peer 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:45.928288 13159 ts_tablet_manager.cc:902] T 8ad236ea5ab347c69026293bef7c04a8 P bb5364d730cb4489a0bfc6c270b7450a: Initiating tablet copy from peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:45.928751 13165 tablet_copy_client.cc:308] T 87652406a8e442e58fe636a85ab72158 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:37231
I20240503 01:59:45.929365 7034 tablet_copy_service.cc:140] P 63331f455be049c1bdd6c23c59ac1bef: Received BeginTabletCopySession request for tablet 87652406a8e442e58fe636a85ab72158 from peer bb5364d730cb4489a0bfc6c270b7450a ({username='slave'} at 127.0.99.134:41551)
I20240503 01:59:45.929483 7034 tablet_copy_service.cc:161] P 63331f455be049c1bdd6c23c59ac1bef: Beginning new tablet copy session on tablet 87652406a8e442e58fe636a85ab72158 from peer bb5364d730cb4489a0bfc6c270b7450a at {username='slave'} at 127.0.99.134:41551: session id = bb5364d730cb4489a0bfc6c270b7450a-87652406a8e442e58fe636a85ab72158
I20240503 01:59:45.930094 13158 tablet_copy_client.cc:308] T 85d3e89fac21402387aa812f754cf5bc P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:37231
I20240503 01:59:45.930445 7034 tablet_copy_source_session.cc:205] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:45.930747 13159 tablet_copy_client.cc:308] T 8ad236ea5ab347c69026293bef7c04a8 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:42943
I20240503 01:59:45.931548 7034 tablet_copy_service.cc:140] P 63331f455be049c1bdd6c23c59ac1bef: Received BeginTabletCopySession request for tablet 85d3e89fac21402387aa812f754cf5bc from peer 12e8c683db054df3b75c6285b98a5ad8 ({username='slave'} at 127.0.99.133:50187)
I20240503 01:59:45.931641 7034 tablet_copy_service.cc:161] P 63331f455be049c1bdd6c23c59ac1bef: Beginning new tablet copy session on tablet 85d3e89fac21402387aa812f754cf5bc from peer 12e8c683db054df3b75c6285b98a5ad8 at {username='slave'} at 127.0.99.133:50187: session id = 12e8c683db054df3b75c6285b98a5ad8-85d3e89fac21402387aa812f754cf5bc
I20240503 01:59:45.931914 13165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87652406a8e442e58fe636a85ab72158. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:45.932641 7034 tablet_copy_source_session.cc:205] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:45.933805 13158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85d3e89fac21402387aa812f754cf5bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:45.935647 13165 tablet_copy_client.cc:783] T 87652406a8e442e58fe636a85ab72158 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:45.935873 13165 tablet_copy_client.cc:647] T 87652406a8e442e58fe636a85ab72158 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:45.937006 13165 tablet_copy_client.cc:523] T 87652406a8e442e58fe636a85ab72158 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:45.938604 13165 tablet_bootstrap.cc:492] T 87652406a8e442e58fe636a85ab72158 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:45.939004 6903 tablet_copy_service.cc:140] P 5dcc33cc621a4cbbae155f3230fb7c46: Received BeginTabletCopySession request for tablet 8ad236ea5ab347c69026293bef7c04a8 from peer bb5364d730cb4489a0bfc6c270b7450a ({username='slave'} at 127.0.99.134:59615)
I20240503 01:59:45.939145 6903 tablet_copy_service.cc:161] P 5dcc33cc621a4cbbae155f3230fb7c46: Beginning new tablet copy session on tablet 8ad236ea5ab347c69026293bef7c04a8 from peer bb5364d730cb4489a0bfc6c270b7450a at {username='slave'} at 127.0.99.134:59615: session id = bb5364d730cb4489a0bfc6c270b7450a-8ad236ea5ab347c69026293bef7c04a8
I20240503 01:59:45.939492 13158 tablet_copy_client.cc:783] T 85d3e89fac21402387aa812f754cf5bc P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:45.939800 13158 tablet_copy_client.cc:647] T 85d3e89fac21402387aa812f754cf5bc P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:45.940201 6903 tablet_copy_source_session.cc:205] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:45.941198 13159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ad236ea5ab347c69026293bef7c04a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:45.943743 13158 tablet_copy_client.cc:523] T 85d3e89fac21402387aa812f754cf5bc P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:45.944540 13159 tablet_copy_client.cc:783] T 8ad236ea5ab347c69026293bef7c04a8 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:45.944830 13159 tablet_copy_client.cc:647] T 8ad236ea5ab347c69026293bef7c04a8 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:45.946097 13159 tablet_copy_client.cc:523] T 8ad236ea5ab347c69026293bef7c04a8 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:45.946308 13158 tablet_bootstrap.cc:492] T 85d3e89fac21402387aa812f754cf5bc P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:45.948094 13159 tablet_bootstrap.cc:492] T 8ad236ea5ab347c69026293bef7c04a8 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:45.961136 13160 tablet_bootstrap.cc:492] T dfdcdd54606e4320a5d0f506eb6dabf8 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:45.961453 13160 tablet_bootstrap.cc:492] T dfdcdd54606e4320a5d0f506eb6dabf8 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap complete.
I20240503 01:59:45.961649 13160 ts_tablet_manager.cc:1367] T dfdcdd54606e4320a5d0f506eb6dabf8 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.056s user 0.026s sys 0.000s
I20240503 01:59:45.962080 13160 raft_consensus.cc:348] T dfdcdd54606e4320a5d0f506eb6dabf8 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:45.962190 13160 raft_consensus.cc:729] T dfdcdd54606e4320a5d0f506eb6dabf8 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:45.962345 13160 consensus_queue.cc:260] T dfdcdd54606e4320a5d0f506eb6dabf8 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:45.964797 13160 ts_tablet_manager.cc:1398] T dfdcdd54606e4320a5d0f506eb6dabf8 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:45.967721 7165 tablet_copy_service.cc:342] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Request end of tablet copy session b7062fbbc2b54fff9e99ec4ea5562c76-dfdcdd54606e4320a5d0f506eb6dabf8 received from {username='slave'} at 127.0.99.135:42797
I20240503 01:59:45.967837 7165 tablet_copy_service.cc:433] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: ending tablet copy session b7062fbbc2b54fff9e99ec4ea5562c76-dfdcdd54606e4320a5d0f506eb6dabf8 on tablet dfdcdd54606e4320a5d0f506eb6dabf8 with peer b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 01:59:45.980953 6733 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:45.981140 6733 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:45.983641 13165 tablet_bootstrap.cc:492] T 87652406a8e442e58fe636a85ab72158 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:45.984158 6733 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 1367d8a2e38844acaa6bf1c57324f6f9
I20240503 01:59:45.984817 9604 tablet_service.cc:1514] Processing DeleteTablet for tablet 1367d8a2e38844acaa6bf1c57324f6f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:45 UTC) from {username='slave'} at 127.0.0.1:58562
I20240503 01:59:45.985080 9472 tablet_service.cc:1514] Processing DeleteTablet for tablet 1367d8a2e38844acaa6bf1c57324f6f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:45 UTC) from {username='slave'} at 127.0.0.1:54678
I20240503 01:59:45.985378 9342 tablet_service.cc:1514] Processing DeleteTablet for tablet 1367d8a2e38844acaa6bf1c57324f6f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:45 UTC) from {username='slave'} at 127.0.0.1:36040
I20240503 01:59:45.985711 13196 tablet_replica.cc:330] T 1367d8a2e38844acaa6bf1c57324f6f9 P 12e8c683db054df3b75c6285b98a5ad8: stopping tablet replica
I20240503 01:59:45.985854 13196 raft_consensus.cc:2229] T 1367d8a2e38844acaa6bf1c57324f6f9 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:45.986016 13196 raft_consensus.cc:2258] T 1367d8a2e38844acaa6bf1c57324f6f9 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:45.986274 13165 tablet_bootstrap.cc:492] T 87652406a8e442e58fe636a85ab72158 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap complete.
I20240503 01:59:45.986546 13165 ts_tablet_manager.cc:1367] T 87652406a8e442e58fe636a85ab72158 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.048s user 0.023s sys 0.003s
I20240503 01:59:45.986735 13196 ts_tablet_manager.cc:1875] T 1367d8a2e38844acaa6bf1c57324f6f9 P 12e8c683db054df3b75c6285b98a5ad8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:45.987129 13165 raft_consensus.cc:348] T 87652406a8e442e58fe636a85ab72158 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:45.987329 13165 raft_consensus.cc:729] T 87652406a8e442e58fe636a85ab72158 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:45.987533 13165 consensus_queue.cc:260] T 87652406a8e442e58fe636a85ab72158 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:45.988452 13159 tablet_bootstrap.cc:492] T 8ad236ea5ab347c69026293bef7c04a8 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:45.988781 13159 tablet_bootstrap.cc:492] T 8ad236ea5ab347c69026293bef7c04a8 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap complete.
I20240503 01:59:45.988945 13165 ts_tablet_manager.cc:1398] T 87652406a8e442e58fe636a85ab72158 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:45.989017 13159 ts_tablet_manager.cc:1367] T 8ad236ea5ab347c69026293bef7c04a8 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.041s user 0.027s sys 0.000s
I20240503 01:59:45.989259 13196 ts_tablet_manager.cc:1888] T 1367d8a2e38844acaa6bf1c57324f6f9 P 12e8c683db054df3b75c6285b98a5ad8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:45.989383 13196 log.cc:1196] T 1367d8a2e38844acaa6bf1c57324f6f9 P 12e8c683db054df3b75c6285b98a5ad8: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/1367d8a2e38844acaa6bf1c57324f6f9
I20240503 01:59:45.989490 7034 tablet_copy_service.cc:342] P 63331f455be049c1bdd6c23c59ac1bef: Request end of tablet copy session bb5364d730cb4489a0bfc6c270b7450a-87652406a8e442e58fe636a85ab72158 received from {username='slave'} at 127.0.99.134:41551
I20240503 01:59:45.989580 7034 tablet_copy_service.cc:433] P 63331f455be049c1bdd6c23c59ac1bef: ending tablet copy session bb5364d730cb4489a0bfc6c270b7450a-87652406a8e442e58fe636a85ab72158 on tablet 87652406a8e442e58fe636a85ab72158 with peer bb5364d730cb4489a0bfc6c270b7450a
I20240503 01:59:45.989661 13159 raft_consensus.cc:348] T 8ad236ea5ab347c69026293bef7c04a8 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:45.989761 13196 ts_tablet_manager.cc:1909] T 1367d8a2e38844acaa6bf1c57324f6f9 P 12e8c683db054df3b75c6285b98a5ad8: Deleting consensus metadata
I20240503 01:59:45.989835 13159 raft_consensus.cc:729] T 8ad236ea5ab347c69026293bef7c04a8 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:45.990073 13159 consensus_queue.cc:260] T 8ad236ea5ab347c69026293bef7c04a8 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:45.990705 6727 catalog_manager.cc:4830] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): tablet 1367d8a2e38844acaa6bf1c57324f6f9 (table rebalancer_dynamic_table_new_name [id=e656a18417874f33aeff28650a93bbe7]) successfully deleted
I20240503 01:59:45.991314 13195 tablet_replica.cc:330] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a: stopping tablet replica
I20240503 01:59:45.991524 13194 tablet_replica.cc:330] T 1367d8a2e38844acaa6bf1c57324f6f9 P b7062fbbc2b54fff9e99ec4ea5562c76: stopping tablet replica
I20240503 01:59:45.991613 13159 ts_tablet_manager.cc:1398] T 8ad236ea5ab347c69026293bef7c04a8 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:45.991731 13195 raft_consensus.cc:2229] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:45.991981 13195 raft_consensus.cc:2258] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:45.992383 6903 tablet_copy_service.cc:342] P 5dcc33cc621a4cbbae155f3230fb7c46: Request end of tablet copy session bb5364d730cb4489a0bfc6c270b7450a-8ad236ea5ab347c69026293bef7c04a8 received from {username='slave'} at 127.0.99.134:59615
I20240503 01:59:45.992467 6903 tablet_copy_service.cc:433] P 5dcc33cc621a4cbbae155f3230fb7c46: ending tablet copy session bb5364d730cb4489a0bfc6c270b7450a-8ad236ea5ab347c69026293bef7c04a8 on tablet 8ad236ea5ab347c69026293bef7c04a8 with peer bb5364d730cb4489a0bfc6c270b7450a
I20240503 01:59:45.993639 13195 ts_tablet_manager.cc:1875] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:45.995780 13194 raft_consensus.cc:2229] T 1367d8a2e38844acaa6bf1c57324f6f9 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:45.995968 13194 raft_consensus.cc:2258] T 1367d8a2e38844acaa6bf1c57324f6f9 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:45.997058 13195 ts_tablet_manager.cc:1888] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:45.997236 13158 tablet_bootstrap.cc:492] T 85d3e89fac21402387aa812f754cf5bc P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:45.997614 13158 tablet_bootstrap.cc:492] T 85d3e89fac21402387aa812f754cf5bc P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap complete.
I20240503 01:59:45.997861 13158 ts_tablet_manager.cc:1367] T 85d3e89fac21402387aa812f754cf5bc P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.052s user 0.029s sys 0.000s
I20240503 01:59:45.998057 13194 ts_tablet_manager.cc:1875] T 1367d8a2e38844acaa6bf1c57324f6f9 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:45.998376 13158 raft_consensus.cc:348] T 85d3e89fac21402387aa812f754cf5bc P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:45.998584 13158 raft_consensus.cc:729] T 85d3e89fac21402387aa812f754cf5bc P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:45.998772 13158 consensus_queue.cc:260] T 85d3e89fac21402387aa812f754cf5bc P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:45.999294 13195 log.cc:1196] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/1367d8a2e38844acaa6bf1c57324f6f9
I20240503 01:59:45.999313 13158 ts_tablet_manager.cc:1398] T 85d3e89fac21402387aa812f754cf5bc P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:45.999872 7034 tablet_copy_service.cc:342] P 63331f455be049c1bdd6c23c59ac1bef: Request end of tablet copy session 12e8c683db054df3b75c6285b98a5ad8-85d3e89fac21402387aa812f754cf5bc received from {username='slave'} at 127.0.99.133:50187
I20240503 01:59:45.999969 7034 tablet_copy_service.cc:433] P 63331f455be049c1bdd6c23c59ac1bef: ending tablet copy session 12e8c683db054df3b75c6285b98a5ad8-85d3e89fac21402387aa812f754cf5bc on tablet 85d3e89fac21402387aa812f754cf5bc with peer 12e8c683db054df3b75c6285b98a5ad8
I20240503 01:59:46.000269 13195 ts_tablet_manager.cc:1909] T 1367d8a2e38844acaa6bf1c57324f6f9 P bb5364d730cb4489a0bfc6c270b7450a: Deleting consensus metadata
I20240503 01:59:46.001102 6725 catalog_manager.cc:4830] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): tablet 1367d8a2e38844acaa6bf1c57324f6f9 (table rebalancer_dynamic_table_new_name [id=e656a18417874f33aeff28650a93bbe7]) successfully deleted
I20240503 01:59:46.002342 13194 ts_tablet_manager.cc:1888] T 1367d8a2e38844acaa6bf1c57324f6f9 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:46.002429 13194 log.cc:1196] T 1367d8a2e38844acaa6bf1c57324f6f9 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/1367d8a2e38844acaa6bf1c57324f6f9
I20240503 01:59:46.002745 13194 ts_tablet_manager.cc:1909] T 1367d8a2e38844acaa6bf1c57324f6f9 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting consensus metadata
I20240503 01:59:46.003582 6727 catalog_manager.cc:4830] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): tablet 1367d8a2e38844acaa6bf1c57324f6f9 (table rebalancer_dynamic_table_new_name [id=e656a18417874f33aeff28650a93bbe7]) successfully deleted
I20240503 01:59:46.007292 13207 ts_tablet_manager.cc:902] T 1b6f98b01ead4cea8f362dc283caa901 P 5dcc33cc621a4cbbae155f3230fb7c46: Initiating tablet copy from peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:46.007871 13207 tablet_copy_client.cc:308] T 1b6f98b01ead4cea8f362dc283caa901 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:37387
I20240503 01:59:46.008530 9383 tablet_copy_service.cc:140] P 12e8c683db054df3b75c6285b98a5ad8: Received BeginTabletCopySession request for tablet 1b6f98b01ead4cea8f362dc283caa901 from peer 5dcc33cc621a4cbbae155f3230fb7c46 ({username='slave'} at 127.0.99.129:37661)
I20240503 01:59:46.008677 9383 tablet_copy_service.cc:161] P 12e8c683db054df3b75c6285b98a5ad8: Beginning new tablet copy session on tablet 1b6f98b01ead4cea8f362dc283caa901 from peer 5dcc33cc621a4cbbae155f3230fb7c46 at {username='slave'} at 127.0.99.129:37661: session id = 5dcc33cc621a4cbbae155f3230fb7c46-1b6f98b01ead4cea8f362dc283caa901
I20240503 01:59:46.010239 9383 tablet_copy_source_session.cc:205] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:46.011718 13207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b6f98b01ead4cea8f362dc283caa901. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.015676 13207 tablet_copy_client.cc:783] T 1b6f98b01ead4cea8f362dc283caa901 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:46.016007 13207 tablet_copy_client.cc:647] T 1b6f98b01ead4cea8f362dc283caa901 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:46.018985 13207 tablet_copy_client.cc:523] T 1b6f98b01ead4cea8f362dc283caa901 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:46.022418 13207 tablet_bootstrap.cc:492] T 1b6f98b01ead4cea8f362dc283caa901 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:46.026895 13232 ts_tablet_manager.cc:902] T 4be54233e31248cd8deafa6b7845a3f8 P db1a0000aaa346f79c0aa4b6c5105191: Initiating tablet copy from peer 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:46.027467 13232 tablet_copy_client.cc:308] T 4be54233e31248cd8deafa6b7845a3f8 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:37231
I20240503 01:59:46.028056 7034 tablet_copy_service.cc:140] P 63331f455be049c1bdd6c23c59ac1bef: Received BeginTabletCopySession request for tablet 4be54233e31248cd8deafa6b7845a3f8 from peer db1a0000aaa346f79c0aa4b6c5105191 ({username='slave'} at 127.0.99.136:38763)
I20240503 01:59:46.028167 7034 tablet_copy_service.cc:161] P 63331f455be049c1bdd6c23c59ac1bef: Beginning new tablet copy session on tablet 4be54233e31248cd8deafa6b7845a3f8 from peer db1a0000aaa346f79c0aa4b6c5105191 at {username='slave'} at 127.0.99.136:38763: session id = db1a0000aaa346f79c0aa4b6c5105191-4be54233e31248cd8deafa6b7845a3f8
I20240503 01:59:46.028446 13207 tablet_bootstrap.cc:492] T 1b6f98b01ead4cea8f362dc283caa901 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:46.028682 13207 tablet_bootstrap.cc:492] T 1b6f98b01ead4cea8f362dc283caa901 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap complete.
I20240503 01:59:46.028833 13207 ts_tablet_manager.cc:1367] T 1b6f98b01ead4cea8f362dc283caa901 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240503 01:59:46.029255 7034 tablet_copy_source_session.cc:205] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:46.030359 13207 raft_consensus.cc:348] T 1b6f98b01ead4cea8f362dc283caa901 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:46.030478 13207 raft_consensus.cc:729] T 1b6f98b01ead4cea8f362dc283caa901 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.030671 13207 consensus_queue.cc:260] T 1b6f98b01ead4cea8f362dc283caa901 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:46.032405 13232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4be54233e31248cd8deafa6b7845a3f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.032732 13207 ts_tablet_manager.cc:1398] T 1b6f98b01ead4cea8f362dc283caa901 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240503 01:59:46.034930 9383 tablet_copy_service.cc:342] P 12e8c683db054df3b75c6285b98a5ad8: Request end of tablet copy session 5dcc33cc621a4cbbae155f3230fb7c46-1b6f98b01ead4cea8f362dc283caa901 received from {username='slave'} at 127.0.99.129:37661
I20240503 01:59:46.035040 9383 tablet_copy_service.cc:433] P 12e8c683db054df3b75c6285b98a5ad8: ending tablet copy session 5dcc33cc621a4cbbae155f3230fb7c46-1b6f98b01ead4cea8f362dc283caa901 on tablet 1b6f98b01ead4cea8f362dc283caa901 with peer 5dcc33cc621a4cbbae155f3230fb7c46
I20240503 01:59:46.036854 12462 consensus_queue.cc:1035] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:46.041857 13232 tablet_copy_client.cc:783] T 4be54233e31248cd8deafa6b7845a3f8 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:46.042551 13044 consensus_queue.cc:1035] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:46.044350 13232 tablet_copy_client.cc:647] T 4be54233e31248cd8deafa6b7845a3f8 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:46.047297 13232 tablet_copy_client.cc:523] T 4be54233e31248cd8deafa6b7845a3f8 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:46.049279 13232 tablet_bootstrap.cc:492] T 4be54233e31248cd8deafa6b7845a3f8 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:46.087085 6738 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:46.092026 7256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:46.097133 13254 ts_tablet_manager.cc:902] T 9cc3435afe5144f9a52a286ac86d234f P 63331f455be049c1bdd6c23c59ac1bef: Initiating tablet copy from peer 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:46.100706 9472 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 01:59:46.101101 13254 tablet_copy_client.cc:308] T 9cc3435afe5144f9a52a286ac86d234f P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36271
I20240503 01:59:46.102371 7295 tablet_copy_service.cc:140] P 1d0e738c34af40fa9950c6f086619b3a: Received BeginTabletCopySession request for tablet 9cc3435afe5144f9a52a286ac86d234f from peer 63331f455be049c1bdd6c23c59ac1bef ({username='slave'} at 127.0.99.130:43003)
I20240503 01:59:46.102492 7295 tablet_copy_service.cc:161] P 1d0e738c34af40fa9950c6f086619b3a: Beginning new tablet copy session on tablet 9cc3435afe5144f9a52a286ac86d234f from peer 63331f455be049c1bdd6c23c59ac1bef at {username='slave'} at 127.0.99.130:43003: session id = 63331f455be049c1bdd6c23c59ac1bef-9cc3435afe5144f9a52a286ac86d234f
I20240503 01:59:46.103449 7295 tablet_copy_source_session.cc:205] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:46.104864 13254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cc3435afe5144f9a52a286ac86d234f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.106079 13232 tablet_bootstrap.cc:492] T 4be54233e31248cd8deafa6b7845a3f8 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:46.106541 13232 tablet_bootstrap.cc:492] T 4be54233e31248cd8deafa6b7845a3f8 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap complete.
I20240503 01:59:46.106810 13232 ts_tablet_manager.cc:1367] T 4be54233e31248cd8deafa6b7845a3f8 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.058s user 0.025s sys 0.004s
I20240503 01:59:46.107381 13232 raft_consensus.cc:348] T 4be54233e31248cd8deafa6b7845a3f8 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:46.107579 13232 raft_consensus.cc:729] T 4be54233e31248cd8deafa6b7845a3f8 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.107759 13232 consensus_queue.cc:260] T 4be54233e31248cd8deafa6b7845a3f8 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:46.109272 13232 ts_tablet_manager.cc:1398] T 4be54233e31248cd8deafa6b7845a3f8 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:46.109860 13254 tablet_copy_client.cc:783] T 9cc3435afe5144f9a52a286ac86d234f P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:46.110131 13254 tablet_copy_client.cc:647] T 9cc3435afe5144f9a52a286ac86d234f P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:46.111030 7034 tablet_copy_service.cc:342] P 63331f455be049c1bdd6c23c59ac1bef: Request end of tablet copy session db1a0000aaa346f79c0aa4b6c5105191-4be54233e31248cd8deafa6b7845a3f8 received from {username='slave'} at 127.0.99.136:38763
I20240503 01:59:46.111109 7034 tablet_copy_service.cc:433] P 63331f455be049c1bdd6c23c59ac1bef: ending tablet copy session db1a0000aaa346f79c0aa4b6c5105191-4be54233e31248cd8deafa6b7845a3f8 on tablet 4be54233e31248cd8deafa6b7845a3f8 with peer db1a0000aaa346f79c0aa4b6c5105191
I20240503 01:59:46.114542 9604 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:46.115278 9472 tablet_service.cc:1467] Processing CreateTablet for tablet 06e485b245f344e5a820973a86ae89a4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6c45beac4ff3468c9ae1763f764ad167]), partition=
I20240503 01:59:46.115540 9472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06e485b245f344e5a820973a86ae89a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.115516 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 06e485b245f344e5a820973a86ae89a4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6c45beac4ff3468c9ae1763f764ad167]), partition=
I20240503 01:59:46.115742 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06e485b245f344e5a820973a86ae89a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.116173 9737 tablet_service.cc:1467] Processing CreateTablet for tablet 06e485b245f344e5a820973a86ae89a4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6c45beac4ff3468c9ae1763f764ad167]), partition=
I20240503 01:59:46.116376 9737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06e485b245f344e5a820973a86ae89a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.116406 13259 ts_tablet_manager.cc:902] T 4e686819aed5407bacc13b3e3a0d4566 P 63331f455be049c1bdd6c23c59ac1bef: Initiating tablet copy from peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:46.116986 13259 tablet_copy_client.cc:308] T 4e686819aed5407bacc13b3e3a0d4566 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:42943
I20240503 01:59:46.117621 6903 tablet_copy_service.cc:140] P 5dcc33cc621a4cbbae155f3230fb7c46: Received BeginTabletCopySession request for tablet 4e686819aed5407bacc13b3e3a0d4566 from peer 63331f455be049c1bdd6c23c59ac1bef ({username='slave'} at 127.0.99.130:58123)
I20240503 01:59:46.117733 6903 tablet_copy_service.cc:161] P 5dcc33cc621a4cbbae155f3230fb7c46: Beginning new tablet copy session on tablet 4e686819aed5407bacc13b3e3a0d4566 from peer 63331f455be049c1bdd6c23c59ac1bef at {username='slave'} at 127.0.99.130:58123: session id = 63331f455be049c1bdd6c23c59ac1bef-4e686819aed5407bacc13b3e3a0d4566
I20240503 01:59:46.118588 13254 tablet_copy_client.cc:523] T 9cc3435afe5144f9a52a286ac86d234f P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:46.118813 6903 tablet_copy_source_session.cc:205] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:46.120607 13265 tablet_bootstrap.cc:492] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:46.121541 13259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e686819aed5407bacc13b3e3a0d4566. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.122987 13254 tablet_bootstrap.cc:492] T 9cc3435afe5144f9a52a286ac86d234f P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:46.130309 6993 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:46.131968 7124 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:46.134388 6861 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:46.136063 9737 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:46.133898 9342 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:46.138263 13266 tablet_bootstrap.cc:492] T 06e485b245f344e5a820973a86ae89a4 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:46.139276 13265 tablet_bootstrap.cc:654] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:46.152173 13266 tablet_bootstrap.cc:654] T 06e485b245f344e5a820973a86ae89a4 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:46.155678 12676 consensus_queue.cc:1035] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:46.159981 13265 tablet_bootstrap.cc:492] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:46.160195 6737 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:46.165071 13265 ts_tablet_manager.cc:1367] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.000s
I20240503 01:59:46.170426 13265 raft_consensus.cc:348] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:46.170725 13265 raft_consensus.cc:374] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:46.170804 13265 raft_consensus.cc:729] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.170993 13265 consensus_queue.cc:260] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:46.171919 13266 tablet_bootstrap.cc:492] T 06e485b245f344e5a820973a86ae89a4 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:46.172227 9737 tablet_service.cc:1467] Processing CreateTablet for tablet 6eeb7c5fa08c48c08c36bf03bd1c85ae (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=5f00a869864944c1a935378f02b67261]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:46.172539 9737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eeb7c5fa08c48c08c36bf03bd1c85ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.172504 9472 tablet_service.cc:1467] Processing CreateTablet for tablet 6eeb7c5fa08c48c08c36bf03bd1c85ae (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=5f00a869864944c1a935378f02b67261]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:46.172798 9472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eeb7c5fa08c48c08c36bf03bd1c85ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.173727 7256 tablet_service.cc:1467] Processing CreateTablet for tablet 6eeb7c5fa08c48c08c36bf03bd1c85ae (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=5f00a869864944c1a935378f02b67261]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:46.174000 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eeb7c5fa08c48c08c36bf03bd1c85ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.174739 13266 ts_tablet_manager.cc:1367] T 06e485b245f344e5a820973a86ae89a4 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.000s
I20240503 01:59:46.175501 13266 raft_consensus.cc:348] T 06e485b245f344e5a820973a86ae89a4 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:46.175691 13266 raft_consensus.cc:374] T 06e485b245f344e5a820973a86ae89a4 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:46.175760 13266 raft_consensus.cc:729] T 06e485b245f344e5a820973a86ae89a4 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.175920 13266 consensus_queue.cc:260] T 06e485b245f344e5a820973a86ae89a4 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:46.176872 13266 ts_tablet_manager.cc:1398] T 06e485b245f344e5a820973a86ae89a4 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:46.176868 13259 tablet_copy_client.cc:783] T 4e686819aed5407bacc13b3e3a0d4566 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:46.177065 13259 tablet_copy_client.cc:647] T 4e686819aed5407bacc13b3e3a0d4566 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:46.177351 13264 tablet_bootstrap.cc:492] T 06e485b245f344e5a820973a86ae89a4 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:46.177527 13265 ts_tablet_manager.cc:1398] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240503 01:59:46.177767 13266 tablet_bootstrap.cc:492] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:46.178699 13264 tablet_bootstrap.cc:654] T 06e485b245f344e5a820973a86ae89a4 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:46.178865 13266 tablet_bootstrap.cc:654] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:46.180102 13269 ts_tablet_manager.cc:902] T fad1113f16ba45009ae409f57eac4550 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Initiating tablet copy from peer db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:46.180873 13269 tablet_copy_client.cc:308] T fad1113f16ba45009ae409f57eac4550 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:42053
I20240503 01:59:46.181394 9777 tablet_copy_service.cc:140] P db1a0000aaa346f79c0aa4b6c5105191: Received BeginTabletCopySession request for tablet fad1113f16ba45009ae409f57eac4550 from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a ({username='slave'} at 127.0.99.131:54483)
I20240503 01:59:46.181491 9777 tablet_copy_service.cc:161] P db1a0000aaa346f79c0aa4b6c5105191: Beginning new tablet copy session on tablet fad1113f16ba45009ae409f57eac4550 from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a at {username='slave'} at 127.0.99.131:54483: session id = c688f3df56ac4b6d9b0c30fa4cc8ca2a-fad1113f16ba45009ae409f57eac4550
I20240503 01:59:46.182384 9777 tablet_copy_source_session.cc:205] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:46.183806 12946 consensus_queue.cc:1035] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:46.184248 13254 tablet_bootstrap.cc:492] T 9cc3435afe5144f9a52a286ac86d234f P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:46.184517 13254 tablet_bootstrap.cc:492] T 9cc3435afe5144f9a52a286ac86d234f P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap complete.
I20240503 01:59:46.184684 13254 ts_tablet_manager.cc:1367] T 9cc3435afe5144f9a52a286ac86d234f P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.062s user 0.004s sys 0.000s
I20240503 01:59:46.185140 13254 raft_consensus.cc:348] T 9cc3435afe5144f9a52a286ac86d234f P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:46.185248 13254 raft_consensus.cc:729] T 9cc3435afe5144f9a52a286ac86d234f P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.185369 13254 consensus_queue.cc:260] T 9cc3435afe5144f9a52a286ac86d234f P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:46.185660 13254 ts_tablet_manager.cc:1398] T 9cc3435afe5144f9a52a286ac86d234f P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:46.188021 13269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fad1113f16ba45009ae409f57eac4550. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.189924 7295 tablet_copy_service.cc:342] P 1d0e738c34af40fa9950c6f086619b3a: Request end of tablet copy session 63331f455be049c1bdd6c23c59ac1bef-9cc3435afe5144f9a52a286ac86d234f received from {username='slave'} at 127.0.99.130:43003
I20240503 01:59:46.190028 7295 tablet_copy_service.cc:433] P 1d0e738c34af40fa9950c6f086619b3a: ending tablet copy session 63331f455be049c1bdd6c23c59ac1bef-9cc3435afe5144f9a52a286ac86d234f on tablet 9cc3435afe5144f9a52a286ac86d234f with peer 63331f455be049c1bdd6c23c59ac1bef
I20240503 01:59:46.190318 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 59db016f885347c8bb7e47f7b6f5f8b3 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=5f00a869864944c1a935378f02b67261]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:46.190621 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59db016f885347c8bb7e47f7b6f5f8b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.191218 9737 tablet_service.cc:1467] Processing CreateTablet for tablet 8feff37a8a6043d2b2f7581deb7c7934 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=5f00a869864944c1a935378f02b67261]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:46.191354 13259 tablet_copy_client.cc:523] T 4e686819aed5407bacc13b3e3a0d4566 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:46.191459 9737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8feff37a8a6043d2b2f7581deb7c7934. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.193048 13259 tablet_bootstrap.cc:492] T 4e686819aed5407bacc13b3e3a0d4566 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:46.194701 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 8feff37a8a6043d2b2f7581deb7c7934 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=5f00a869864944c1a935378f02b67261]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:46.194819 9605 tablet_service.cc:1467] Processing CreateTablet for tablet 8feff37a8a6043d2b2f7581deb7c7934 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=5f00a869864944c1a935378f02b67261]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:46.194857 9604 tablet_service.cc:1467] Processing CreateTablet for tablet 59db016f885347c8bb7e47f7b6f5f8b3 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=5f00a869864944c1a935378f02b67261]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:46.194965 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8feff37a8a6043d2b2f7581deb7c7934. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.195053 9605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8feff37a8a6043d2b2f7581deb7c7934. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.195354 7124 tablet_service.cc:1467] Processing CreateTablet for tablet 59db016f885347c8bb7e47f7b6f5f8b3 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=5f00a869864944c1a935378f02b67261]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:46.195629 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59db016f885347c8bb7e47f7b6f5f8b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.197126 12873 consensus_queue.cc:1035] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:46.198872 13265 tablet_bootstrap.cc:492] T 59db016f885347c8bb7e47f7b6f5f8b3 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:46.198977 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59db016f885347c8bb7e47f7b6f5f8b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.201694 13265 tablet_bootstrap.cc:654] T 59db016f885347c8bb7e47f7b6f5f8b3 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:46.202067 13266 tablet_bootstrap.cc:492] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:46.202224 13266 ts_tablet_manager.cc:1367] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240503 01:59:46.202780 13266 raft_consensus.cc:348] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:46.202973 13266 raft_consensus.cc:374] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:46.203051 13266 raft_consensus.cc:729] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.203266 13266 consensus_queue.cc:260] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:46.204398 13264 tablet_bootstrap.cc:492] T 06e485b245f344e5a820973a86ae89a4 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:46.204517 13264 ts_tablet_manager.cc:1367] T 06e485b245f344e5a820973a86ae89a4 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20240503 01:59:46.204520 13266 ts_tablet_manager.cc:1398] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:46.204895 13266 tablet_bootstrap.cc:492] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:46.205171 13264 raft_consensus.cc:348] T 06e485b245f344e5a820973a86ae89a4 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:46.205299 13264 raft_consensus.cc:374] T 06e485b245f344e5a820973a86ae89a4 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:46.205336 13264 raft_consensus.cc:729] T 06e485b245f344e5a820973a86ae89a4 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.205474 13264 consensus_queue.cc:260] T 06e485b245f344e5a820973a86ae89a4 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:46.205806 13264 ts_tablet_manager.cc:1398] T 06e485b245f344e5a820973a86ae89a4 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:46.206015 13264 tablet_bootstrap.cc:492] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:46.206482 13266 tablet_bootstrap.cc:654] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:46.207608 13264 tablet_bootstrap.cc:654] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:46.208464 13272 tablet_bootstrap.cc:492] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:46.209519 13272 tablet_bootstrap.cc:654] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:46.210156 13283 tablet_bootstrap.cc:492] T 8feff37a8a6043d2b2f7581deb7c7934 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:46.211189 13283 tablet_bootstrap.cc:654] T 8feff37a8a6043d2b2f7581deb7c7934 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:46.211414 13264 tablet_bootstrap.cc:492] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:46.211508 13264 ts_tablet_manager.cc:1367] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:46.211881 13284 tablet_bootstrap.cc:492] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:46.211984 13264 raft_consensus.cc:348] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:46.212093 13264 raft_consensus.cc:374] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:46.212119 13264 raft_consensus.cc:729] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.212242 13264 consensus_queue.cc:260] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:46.212540 13264 ts_tablet_manager.cc:1398] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:46.213057 13284 tablet_bootstrap.cc:654] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:46.213863 13272 tablet_bootstrap.cc:492] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:46.214025 13272 ts_tablet_manager.cc:1367] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:46.214690 13272 raft_consensus.cc:348] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:46.214802 13272 raft_consensus.cc:374] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:46.214838 13272 raft_consensus.cc:729] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.214996 13272 consensus_queue.cc:260] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:46.215371 13272 ts_tablet_manager.cc:1398] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:46.223842 13265 tablet_bootstrap.cc:492] T 59db016f885347c8bb7e47f7b6f5f8b3 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:46.224054 13265 ts_tablet_manager.cc:1367] T 59db016f885347c8bb7e47f7b6f5f8b3 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20240503 01:59:46.224699 13259 tablet_bootstrap.cc:492] T 4e686819aed5407bacc13b3e3a0d4566 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:46.224639 13265 raft_consensus.cc:348] T 59db016f885347c8bb7e47f7b6f5f8b3 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:46.224764 13265 raft_consensus.cc:374] T 59db016f885347c8bb7e47f7b6f5f8b3 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:46.224813 13265 raft_consensus.cc:729] T 59db016f885347c8bb7e47f7b6f5f8b3 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.224983 13259 tablet_bootstrap.cc:492] T 4e686819aed5407bacc13b3e3a0d4566 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap complete.
I20240503 01:59:46.224952 13265 consensus_queue.cc:260] T 59db016f885347c8bb7e47f7b6f5f8b3 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:46.225154 13259 ts_tablet_manager.cc:1367] T 4e686819aed5407bacc13b3e3a0d4566 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.000s
I20240503 01:59:46.225704 13259 raft_consensus.cc:348] T 4e686819aed5407bacc13b3e3a0d4566 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:46.225818 13259 raft_consensus.cc:729] T 4e686819aed5407bacc13b3e3a0d4566 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.225944 13259 consensus_queue.cc:260] T 4e686819aed5407bacc13b3e3a0d4566 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:46.226258 13259 ts_tablet_manager.cc:1398] T 4e686819aed5407bacc13b3e3a0d4566 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:46.226292 13265 ts_tablet_manager.cc:1398] T 59db016f885347c8bb7e47f7b6f5f8b3 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:46.226294 12873 consensus_queue.cc:1035] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:46.226495 13265 tablet_bootstrap.cc:492] T 8feff37a8a6043d2b2f7581deb7c7934 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:46.227102 6903 tablet_copy_service.cc:342] P 5dcc33cc621a4cbbae155f3230fb7c46: Request end of tablet copy session 63331f455be049c1bdd6c23c59ac1bef-4e686819aed5407bacc13b3e3a0d4566 received from {username='slave'} at 127.0.99.130:58123
I20240503 01:59:46.227181 6903 tablet_copy_service.cc:433] P 5dcc33cc621a4cbbae155f3230fb7c46: ending tablet copy session 63331f455be049c1bdd6c23c59ac1bef-4e686819aed5407bacc13b3e3a0d4566 on tablet 4e686819aed5407bacc13b3e3a0d4566 with peer 63331f455be049c1bdd6c23c59ac1bef
I20240503 01:59:46.230722 13269 tablet_copy_client.cc:783] T fad1113f16ba45009ae409f57eac4550 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:46.231011 13269 tablet_copy_client.cc:647] T fad1113f16ba45009ae409f57eac4550 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:46.233989 13265 tablet_bootstrap.cc:654] T 8feff37a8a6043d2b2f7581deb7c7934 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:46.234951 13269 tablet_copy_client.cc:523] T fad1113f16ba45009ae409f57eac4550 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:46.235101 13265 tablet_bootstrap.cc:492] T 8feff37a8a6043d2b2f7581deb7c7934 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:46.235462 13265 ts_tablet_manager.cc:1367] T 8feff37a8a6043d2b2f7581deb7c7934 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240503 01:59:46.236013 13265 raft_consensus.cc:348] T 8feff37a8a6043d2b2f7581deb7c7934 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:46.236178 13265 raft_consensus.cc:374] T 8feff37a8a6043d2b2f7581deb7c7934 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:46.236258 13265 raft_consensus.cc:729] T 8feff37a8a6043d2b2f7581deb7c7934 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.236418 13265 consensus_queue.cc:260] T 8feff37a8a6043d2b2f7581deb7c7934 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:46.236816 13265 ts_tablet_manager.cc:1398] T 8feff37a8a6043d2b2f7581deb7c7934 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:46.238893 13266 tablet_bootstrap.cc:492] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:46.239001 13266 ts_tablet_manager.cc:1367] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.001s
I20240503 01:59:46.239459 13266 raft_consensus.cc:348] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:46.239560 13266 raft_consensus.cc:374] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:46.239601 13266 raft_consensus.cc:729] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.239727 13266 consensus_queue.cc:260] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:46.239985 13266 ts_tablet_manager.cc:1398] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:46.240188 13275 ts_tablet_manager.cc:902] T 84812ece4f27484eb008bfbebdbfee84 P 1d0e738c34af40fa9950c6f086619b3a: Initiating tablet copy from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:46.241670 13232 ts_tablet_manager.cc:902] T 79efb54e1f8143be868e9ef37f5f925d P db1a0000aaa346f79c0aa4b6c5105191: Initiating tablet copy from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:46.242178 13232 tablet_copy_client.cc:308] T 79efb54e1f8143be868e9ef37f5f925d P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41979
I20240503 01:59:46.243203 7165 tablet_copy_service.cc:140] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Received BeginTabletCopySession request for tablet 79efb54e1f8143be868e9ef37f5f925d from peer db1a0000aaa346f79c0aa4b6c5105191 ({username='slave'} at 127.0.99.136:40777)
I20240503 01:59:46.243304 7165 tablet_copy_service.cc:161] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Beginning new tablet copy session on tablet 79efb54e1f8143be868e9ef37f5f925d from peer db1a0000aaa346f79c0aa4b6c5105191 at {username='slave'} at 127.0.99.136:40777: session id = db1a0000aaa346f79c0aa4b6c5105191-79efb54e1f8143be868e9ef37f5f925d
I20240503 01:59:46.244246 7165 tablet_copy_source_session.cc:205] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:46.247197 13232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79efb54e1f8143be868e9ef37f5f925d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.249961 13232 tablet_copy_client.cc:783] T 79efb54e1f8143be868e9ef37f5f925d P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:46.250226 13232 tablet_copy_client.cc:647] T 79efb54e1f8143be868e9ef37f5f925d P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:46.251775 13269 tablet_bootstrap.cc:492] T fad1113f16ba45009ae409f57eac4550 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:46.254335 13275 tablet_copy_client.cc:308] T 84812ece4f27484eb008bfbebdbfee84 P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41979
I20240503 01:59:46.254911 7165 tablet_copy_service.cc:140] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Received BeginTabletCopySession request for tablet 84812ece4f27484eb008bfbebdbfee84 from peer 1d0e738c34af40fa9950c6f086619b3a ({username='slave'} at 127.0.99.132:38863)
I20240503 01:59:46.255012 7165 tablet_copy_service.cc:161] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Beginning new tablet copy session on tablet 84812ece4f27484eb008bfbebdbfee84 from peer 1d0e738c34af40fa9950c6f086619b3a at {username='slave'} at 127.0.99.132:38863: session id = 1d0e738c34af40fa9950c6f086619b3a-84812ece4f27484eb008bfbebdbfee84
I20240503 01:59:46.256062 7165 tablet_copy_source_session.cc:205] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:46.257390 13275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84812ece4f27484eb008bfbebdbfee84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:46.259781 13275 tablet_copy_client.cc:783] T 84812ece4f27484eb008bfbebdbfee84 P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:46.259945 13275 tablet_copy_client.cc:647] T 84812ece4f27484eb008bfbebdbfee84 P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:46.260711 13283 tablet_bootstrap.cc:492] T 8feff37a8a6043d2b2f7581deb7c7934 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:46.260816 13283 ts_tablet_manager.cc:1367] T 8feff37a8a6043d2b2f7581deb7c7934 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.051s user 0.002s sys 0.000s
I20240503 01:59:46.260946 13284 tablet_bootstrap.cc:492] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:46.261032 13284 ts_tablet_manager.cc:1367] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.000s
I20240503 01:59:46.261343 13283 raft_consensus.cc:348] T 8feff37a8a6043d2b2f7581deb7c7934 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:46.261466 13283 raft_consensus.cc:374] T 8feff37a8a6043d2b2f7581deb7c7934 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:46.261514 13283 raft_consensus.cc:729] T 8feff37a8a6043d2b2f7581deb7c7934 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.261459 13284 raft_consensus.cc:348] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:46.261579 13284 raft_consensus.cc:374] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:46.261608 13284 raft_consensus.cc:729] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.261653 13283 consensus_queue.cc:260] T 8feff37a8a6043d2b2f7581deb7c7934 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:46.261732 13284 consensus_queue.cc:260] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:46.261786 13232 tablet_copy_client.cc:523] T 79efb54e1f8143be868e9ef37f5f925d P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:46.261963 13283 ts_tablet_manager.cc:1398] T 8feff37a8a6043d2b2f7581deb7c7934 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:46.261976 13284 ts_tablet_manager.cc:1398] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:46.262126 13283 tablet_bootstrap.cc:492] T 59db016f885347c8bb7e47f7b6f5f8b3 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:46.263373 13283 tablet_bootstrap.cc:654] T 59db016f885347c8bb7e47f7b6f5f8b3 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:46.263387 13232 tablet_bootstrap.cc:492] T 79efb54e1f8143be868e9ef37f5f925d P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:46.263494 9363 raft_consensus.cc:1203] T 85d3e89fac21402387aa812f754cf5bc P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:46.264731 12070 raft_consensus.cc:482] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:46.264835 12070 raft_consensus.cc:504] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:46.265228 12070 leader_election.cc:288] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:46.266425 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeb7c5fa08c48c08c36bf03bd1c85ae" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:46.266621 9757 raft_consensus.cc:2454] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:46.266914 9428 leader_election.cc:302] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:46.267093 12070 raft_consensus.cc:2781] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:46.267140 12070 raft_consensus.cc:482] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:46.267176 12070 raft_consensus.cc:3037] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:46.267653 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeb7c5fa08c48c08c36bf03bd1c85ae" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:46.267802 7274 raft_consensus.cc:2454] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:46.268247 12070 raft_consensus.cc:504] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:46.268589 12070 leader_election.cc:288] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Requested vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:46.269953 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeb7c5fa08c48c08c36bf03bd1c85ae" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:46.270071 9755 raft_consensus.cc:3037] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:46.271101 9755 raft_consensus.cc:2454] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:46.271880 9428 leader_election.cc:302] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:46.272300 12070 raft_consensus.cc:2781] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:46.272394 12070 raft_consensus.cc:686] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Becoming Leader. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Running, Role: LEADER
I20240503 01:59:46.272562 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeb7c5fa08c48c08c36bf03bd1c85ae" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:46.272532 12070 consensus_queue.cc:237] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:46.272670 7274 raft_consensus.cc:3037] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:46.273437 13275 tablet_copy_client.cc:523] T 84812ece4f27484eb008bfbebdbfee84 P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:46.273839 7274 raft_consensus.cc:2454] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:46.276145 13275 tablet_bootstrap.cc:492] T 84812ece4f27484eb008bfbebdbfee84 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:46.276736 6735 catalog_manager.cc:5484] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: term changed from 0 to 1, leader changed from <none> to bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:46.280577 13283 tablet_bootstrap.cc:492] T 59db016f885347c8bb7e47f7b6f5f8b3 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:46.280711 13283 ts_tablet_manager.cc:1367] T 59db016f885347c8bb7e47f7b6f5f8b3 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.002s
I20240503 01:59:46.281231 13283 raft_consensus.cc:348] T 59db016f885347c8bb7e47f7b6f5f8b3 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:46.281366 13283 raft_consensus.cc:374] T 59db016f885347c8bb7e47f7b6f5f8b3 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:46.281402 13283 raft_consensus.cc:729] T 59db016f885347c8bb7e47f7b6f5f8b3 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.281546 13283 consensus_queue.cc:260] T 59db016f885347c8bb7e47f7b6f5f8b3 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:46.281849 13283 ts_tablet_manager.cc:1398] T 59db016f885347c8bb7e47f7b6f5f8b3 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:46.286687 13275 tablet_bootstrap.cc:492] T 84812ece4f27484eb008bfbebdbfee84 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:46.287081 13275 tablet_bootstrap.cc:492] T 84812ece4f27484eb008bfbebdbfee84 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap complete.
I20240503 01:59:46.287101 13269 tablet_bootstrap.cc:492] T fad1113f16ba45009ae409f57eac4550 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:46.287292 13275 ts_tablet_manager.cc:1367] T 84812ece4f27484eb008bfbebdbfee84 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.004s
I20240503 01:59:46.287772 13275 raft_consensus.cc:348] T 84812ece4f27484eb008bfbebdbfee84 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:46.287901 13275 raft_consensus.cc:729] T 84812ece4f27484eb008bfbebdbfee84 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.288079 13275 consensus_queue.cc:260] T 84812ece4f27484eb008bfbebdbfee84 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:46.288555 13275 ts_tablet_manager.cc:1398] T 84812ece4f27484eb008bfbebdbfee84 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:46.289079 7165 tablet_copy_service.cc:342] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Request end of tablet copy session 1d0e738c34af40fa9950c6f086619b3a-84812ece4f27484eb008bfbebdbfee84 received from {username='slave'} at 127.0.99.132:38863
I20240503 01:59:46.289150 7165 tablet_copy_service.cc:433] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: ending tablet copy session 1d0e738c34af40fa9950c6f086619b3a-84812ece4f27484eb008bfbebdbfee84 on tablet 84812ece4f27484eb008bfbebdbfee84 with peer 1d0e738c34af40fa9950c6f086619b3a
I20240503 01:59:46.292141 13269 tablet_bootstrap.cc:492] T fad1113f16ba45009ae409f57eac4550 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap complete.
I20240503 01:59:46.292598 13269 ts_tablet_manager.cc:1367] T fad1113f16ba45009ae409f57eac4550 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.000s
I20240503 01:59:46.293154 13269 raft_consensus.cc:348] T fad1113f16ba45009ae409f57eac4550 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:46.293282 13269 raft_consensus.cc:729] T fad1113f16ba45009ae409f57eac4550 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.293453 13269 consensus_queue.cc:260] T fad1113f16ba45009ae409f57eac4550 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:46.293866 13269 ts_tablet_manager.cc:1398] T fad1113f16ba45009ae409f57eac4550 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:46.294559 9777 tablet_copy_service.cc:342] P db1a0000aaa346f79c0aa4b6c5105191: Request end of tablet copy session c688f3df56ac4b6d9b0c30fa4cc8ca2a-fad1113f16ba45009ae409f57eac4550 received from {username='slave'} at 127.0.99.131:54483
I20240503 01:59:46.294646 9777 tablet_copy_service.cc:433] P db1a0000aaa346f79c0aa4b6c5105191: ending tablet copy session c688f3df56ac4b6d9b0c30fa4cc8ca2a-fad1113f16ba45009ae409f57eac4550 on tablet fad1113f16ba45009ae409f57eac4550 with peer c688f3df56ac4b6d9b0c30fa4cc8ca2a
I20240503 01:59:46.313338 13232 tablet_bootstrap.cc:492] T 79efb54e1f8143be868e9ef37f5f925d P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:46.313709 13232 tablet_bootstrap.cc:492] T 79efb54e1f8143be868e9ef37f5f925d P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap complete.
I20240503 01:59:46.313894 13232 ts_tablet_manager.cc:1367] T 79efb54e1f8143be868e9ef37f5f925d P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.051s user 0.027s sys 0.002s
I20240503 01:59:46.314371 13232 raft_consensus.cc:348] T 79efb54e1f8143be868e9ef37f5f925d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:46.314500 13232 raft_consensus.cc:729] T 79efb54e1f8143be868e9ef37f5f925d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:46.314661 13232 consensus_queue.cc:260] T 79efb54e1f8143be868e9ef37f5f925d P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:46.314935 13232 ts_tablet_manager.cc:1398] T 79efb54e1f8143be868e9ef37f5f925d P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:46.315616 7165 tablet_copy_service.cc:342] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Request end of tablet copy session db1a0000aaa346f79c0aa4b6c5105191-79efb54e1f8143be868e9ef37f5f925d received from {username='slave'} at 127.0.99.136:40777
I20240503 01:59:46.315708 7165 tablet_copy_service.cc:433] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: ending tablet copy session db1a0000aaa346f79c0aa4b6c5105191-79efb54e1f8143be868e9ef37f5f925d on tablet 79efb54e1f8143be868e9ef37f5f925d with peer db1a0000aaa346f79c0aa4b6c5105191
I20240503 01:59:46.326875 9494 raft_consensus.cc:1203] T 87652406a8e442e58fe636a85ab72158 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:46.405074 9625 raft_consensus.cc:1203] T dfdcdd54606e4320a5d0f506eb6dabf8 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:46.415999 9494 raft_consensus.cc:1203] T 8ad236ea5ab347c69026293bef7c04a8 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:46.454159 9755 raft_consensus.cc:1203] T 4be54233e31248cd8deafa6b7845a3f8 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:46.498095 13349 raft_consensus.cc:482] T 59db016f885347c8bb7e47f7b6f5f8b3 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:46.498240 13349 raft_consensus.cc:504] T 59db016f885347c8bb7e47f7b6f5f8b3 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:46.498991 13349 leader_election.cc:288] T 59db016f885347c8bb7e47f7b6f5f8b3 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:46.499398 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59db016f885347c8bb7e47f7b6f5f8b3" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:46.499430 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59db016f885347c8bb7e47f7b6f5f8b3" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:46.499565 7143 raft_consensus.cc:2454] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:46.499576 9625 raft_consensus.cc:2454] T 59db016f885347c8bb7e47f7b6f5f8b3 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:46.499964 9298 leader_election.cc:302] T 59db016f885347c8bb7e47f7b6f5f8b3 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:46.500697 13349 raft_consensus.cc:2781] T 59db016f885347c8bb7e47f7b6f5f8b3 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:46.500784 13349 raft_consensus.cc:482] T 59db016f885347c8bb7e47f7b6f5f8b3 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:46.500836 13349 raft_consensus.cc:3037] T 59db016f885347c8bb7e47f7b6f5f8b3 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:46.501981 13349 raft_consensus.cc:504] T 59db016f885347c8bb7e47f7b6f5f8b3 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:46.502446 13349 leader_election.cc:288] T 59db016f885347c8bb7e47f7b6f5f8b3 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:46.502890 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59db016f885347c8bb7e47f7b6f5f8b3" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:46.502899 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59db016f885347c8bb7e47f7b6f5f8b3" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:46.503015 7143 raft_consensus.cc:3037] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:46.503015 9625 raft_consensus.cc:3037] T 59db016f885347c8bb7e47f7b6f5f8b3 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:46.504034 7143 raft_consensus.cc:2454] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:46.504034 9625 raft_consensus.cc:2454] T 59db016f885347c8bb7e47f7b6f5f8b3 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:46.504397 9298 leader_election.cc:302] T 59db016f885347c8bb7e47f7b6f5f8b3 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:46.504815 13349 raft_consensus.cc:2781] T 59db016f885347c8bb7e47f7b6f5f8b3 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:46.504909 13349 raft_consensus.cc:686] T 59db016f885347c8bb7e47f7b6f5f8b3 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Running, Role: LEADER
I20240503 01:59:46.505031 13349 consensus_queue.cc:237] T 59db016f885347c8bb7e47f7b6f5f8b3 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:46.507472 6737 catalog_manager.cc:5484] T 59db016f885347c8bb7e47f7b6f5f8b3 P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:46.512868 6882 raft_consensus.cc:1203] T 1b6f98b01ead4cea8f362dc283caa901 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 01:59:46.522047 7012 raft_consensus.cc:1203] T 4e686819aed5407bacc13b3e3a0d4566 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 01:59:46.526747 13044 raft_consensus.cc:482] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:46.526934 13044 raft_consensus.cc:504] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:46.527487 13044 leader_election.cc:288] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:46.528061 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feff37a8a6043d2b2f7581deb7c7934" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:46.528020 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feff37a8a6043d2b2f7581deb7c7934" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:46.528231 9363 raft_consensus.cc:2454] T 8feff37a8a6043d2b2f7581deb7c7934 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 01:59:46.528231 9625 raft_consensus.cc:2454] T 8feff37a8a6043d2b2f7581deb7c7934 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 01:59:46.528565 9692 leader_election.cc:302] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:46.528789 13044 raft_consensus.cc:2781] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:46.528869 13044 raft_consensus.cc:482] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:46.528913 13044 raft_consensus.cc:3037] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:46.529829 13044 raft_consensus.cc:504] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:46.530175 13044 leader_election.cc:288] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:46.530838 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feff37a8a6043d2b2f7581deb7c7934" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:46.530856 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feff37a8a6043d2b2f7581deb7c7934" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:46.530978 9625 raft_consensus.cc:3037] T 8feff37a8a6043d2b2f7581deb7c7934 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:46.530983 9363 raft_consensus.cc:3037] T 8feff37a8a6043d2b2f7581deb7c7934 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:46.531816 9625 raft_consensus.cc:2454] T 8feff37a8a6043d2b2f7581deb7c7934 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:46.532022 9363 raft_consensus.cc:2454] T 8feff37a8a6043d2b2f7581deb7c7934 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:46.532195 9692 leader_election.cc:302] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:46.532377 13044 raft_consensus.cc:2781] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:46.532475 13044 raft_consensus.cc:686] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Becoming Leader. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Running, Role: LEADER
I20240503 01:59:46.532615 13044 consensus_queue.cc:237] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:46.535038 6736 catalog_manager.cc:5484] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: term changed from 0 to 1, leader changed from <none> to db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:46.583201 13349 raft_consensus.cc:482] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:46.583330 13349 raft_consensus.cc:504] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:46.583745 13349 leader_election.cc:288] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:46.584172 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e485b245f344e5a820973a86ae89a4" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:46.584167 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e485b245f344e5a820973a86ae89a4" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:46.584300 9494 raft_consensus.cc:2454] T 06e485b245f344e5a820973a86ae89a4 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:46.584309 9755 raft_consensus.cc:2454] T 06e485b245f344e5a820973a86ae89a4 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:46.584626 9296 leader_election.cc:302] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:46.584848 13349 raft_consensus.cc:2781] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:46.584897 13349 raft_consensus.cc:482] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:46.584931 13349 raft_consensus.cc:3037] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:46.585988 13349 raft_consensus.cc:504] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:46.586359 13349 leader_election.cc:288] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Requested vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:46.586786 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e485b245f344e5a820973a86ae89a4" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:46.586923 9755 raft_consensus.cc:3037] T 06e485b245f344e5a820973a86ae89a4 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:46.587036 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e485b245f344e5a820973a86ae89a4" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:46.587174 9494 raft_consensus.cc:3037] T 06e485b245f344e5a820973a86ae89a4 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:46.587890 9755 raft_consensus.cc:2454] T 06e485b245f344e5a820973a86ae89a4 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:46.587982 9494 raft_consensus.cc:2454] T 06e485b245f344e5a820973a86ae89a4 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:46.588203 9297 leader_election.cc:302] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:46.588575 13349 raft_consensus.cc:2781] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:46.588757 13349 raft_consensus.cc:686] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Running, Role: LEADER
I20240503 01:59:46.588943 13349 consensus_queue.cc:237] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:46.591956 6736 catalog_manager.cc:5484] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:46.612748 6861 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:46.613363 7256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:46.615233 9472 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:46.627839 6993 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:46.627786 7012 raft_consensus.cc:1203] T 9cc3435afe5144f9a52a286ac86d234f P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 01:59:46.631788 7124 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:46.632126 9604 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:46.633123 9342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:46.651897 9737 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:46.697958 12070 consensus_queue.cc:1035] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:46.701308 13101 consensus_queue.cc:1035] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:46.735188 6736 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:46.742283 9755 raft_consensus.cc:1261] T 06e485b245f344e5a820973a86ae89a4 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:46.742208 7145 raft_consensus.cc:1203] T fad1113f16ba45009ae409f57eac4550 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 01:59:46.743171 9494 raft_consensus.cc:1261] T 06e485b245f344e5a820973a86ae89a4 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:46.743670 13349 consensus_queue.cc:1035] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:46.743880 13010 consensus_queue.cc:1035] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:46.746958 13399 tablet.cc:1722] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:46.747404 13399 tablet.cc:1620] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 01:59:46.747526 6736 catalog_manager.cc:5576] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471) does not have the latest schema for tablet 06e485b245f344e5a820973a86ae89a4 (table rebalancer_dynamic_table [id=6c45beac4ff3468c9ae1763f764ad167]). Expected version 1 got 0
I20240503 01:59:46.748143 13400 tablet.cc:1722] T 06e485b245f344e5a820973a86ae89a4 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:46.748586 6737 catalog_manager.cc:5576] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053) does not have the latest schema for tablet 06e485b245f344e5a820973a86ae89a4 (table rebalancer_dynamic_table [id=6c45beac4ff3468c9ae1763f764ad167]). Expected version 1 got 0
I20240503 01:59:46.748601 13400 tablet.cc:1620] T 06e485b245f344e5a820973a86ae89a4 P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:46.751818 9755 raft_consensus.cc:1203] T 79efb54e1f8143be868e9ef37f5f925d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:46.754242 6736 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=6c45beac4ff3468c9ae1763f764ad167] alter complete (version 1)
I20240503 01:59:46.760658 6736 catalog_manager.cc:5576] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471) does not have the latest schema for tablet 06e485b245f344e5a820973a86ae89a4 (table rebalancer_dynamic_table [id=6c45beac4ff3468c9ae1763f764ad167]). Expected version 1 got 0
I20240503 01:59:46.760740 13401 tablet.cc:1722] T 06e485b245f344e5a820973a86ae89a4 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:46.761219 13401 tablet.cc:1620] T 06e485b245f344e5a820973a86ae89a4 P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:46.772065 7274 raft_consensus.cc:1203] T 84812ece4f27484eb008bfbebdbfee84 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 01:59:46.837428 7256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:46.843662 9342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:46.844967 9604 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:46.845435 6861 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:46.858330 9472 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 01:59:46.858896 9737 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:46.860376 6993 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:46.861069 6739 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:46.864019 7124 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:46.882089 13399 tablet.cc:1722] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:46.882475 13399 tablet.cc:1620] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 01:59:46.882565 13400 tablet.cc:1722] T 06e485b245f344e5a820973a86ae89a4 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:46.882833 13400 tablet.cc:1620] T 06e485b245f344e5a820973a86ae89a4 P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:46.892596 6736 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=6c45beac4ff3468c9ae1763f764ad167] alter complete (version 2)
I20240503 01:59:46.896034 13401 tablet.cc:1722] T 06e485b245f344e5a820973a86ae89a4 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:46.896421 13401 tablet.cc:1620] T 06e485b245f344e5a820973a86ae89a4 P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:46.942410 13010 consensus_queue.cc:1035] T 59db016f885347c8bb7e47f7b6f5f8b3 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:46.947631 13010 consensus_queue.cc:1035] T 59db016f885347c8bb7e47f7b6f5f8b3 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:46.993664 9363 consensus_queue.cc:237] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:46.995527 9755 raft_consensus.cc:1261] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:46.995657 6882 raft_consensus.cc:1261] T 1b6f98b01ead4cea8f362dc283caa901 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:46.996037 13010 consensus_queue.cc:1035] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:46.996229 13349 consensus_queue.cc:1035] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:46.997417 6736 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:46.998191 13010 raft_consensus.cc:2932] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:46.999030 6882 raft_consensus.cc:2932] T 1b6f98b01ead4cea8f362dc283caa901 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:46.999867 9755 raft_consensus.cc:2932] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:47.001957 6739 catalog_manager.cc:5484] T 1b6f98b01ead4cea8f362dc283caa901 P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:47.004050 12462 consensus_queue.cc:1035] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:47.005285 13399 tablet.cc:1722] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:47.007838 9472 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b6f98b01ead4cea8f362dc283caa901 with delete_type TABLET_DATA_TOMBSTONED (TS bb5364d730cb4489a0bfc6c270b7450a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54678
I20240503 01:59:47.008016 13401 tablet.cc:1722] T 06e485b245f344e5a820973a86ae89a4 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:47.008420 13459 tablet_replica.cc:330] T 1b6f98b01ead4cea8f362dc283caa901 P bb5364d730cb4489a0bfc6c270b7450a: stopping tablet replica
I20240503 01:59:47.008635 13459 raft_consensus.cc:2229] T 1b6f98b01ead4cea8f362dc283caa901 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:47.008842 13459 raft_consensus.cc:2258] T 1b6f98b01ead4cea8f362dc283caa901 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:47.009589 13459 ts_tablet_manager.cc:1875] T 1b6f98b01ead4cea8f362dc283caa901 P bb5364d730cb4489a0bfc6c270b7450a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:47.011258 13044 consensus_queue.cc:1035] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:47.014151 13459 ts_tablet_manager.cc:1888] T 1b6f98b01ead4cea8f362dc283caa901 P bb5364d730cb4489a0bfc6c270b7450a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 01:59:47.014513 13459 log.cc:1196] T 1b6f98b01ead4cea8f362dc283caa901 P bb5364d730cb4489a0bfc6c270b7450a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/1b6f98b01ead4cea8f362dc283caa901
I20240503 01:59:47.014420 13400 tablet.cc:1722] T 06e485b245f344e5a820973a86ae89a4 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:47.019587 6725 catalog_manager.cc:4830] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): tablet 1b6f98b01ead4cea8f362dc283caa901 (table rebalancer_extra_table_8 [id=5094b51906a7434daee33202abd81bfc]) successfully deleted
I20240503 01:59:47.023363 6737 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=6c45beac4ff3468c9ae1763f764ad167] alter complete (version 3)
I20240503 01:59:47.085204 6736 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:47.100833 9472 tablet_service.cc:1467] Processing CreateTablet for tablet df13afe0f90c4c24b66026048b5852ec (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=83830de54936404aa798ca5e4773319d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.101538 9472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df13afe0f90c4c24b66026048b5852ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.102257 9604 tablet_service.cc:1467] Processing CreateTablet for tablet df13afe0f90c4c24b66026048b5852ec (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=83830de54936404aa798ca5e4773319d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.102335 6861 tablet_service.cc:1467] Processing CreateTablet for tablet f454ed0b1a924bec9d08f8740137be9b (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=83830de54936404aa798ca5e4773319d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.102640 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df13afe0f90c4c24b66026048b5852ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.102710 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f454ed0b1a924bec9d08f8740137be9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.104602 9342 tablet_service.cc:1467] Processing CreateTablet for tablet df13afe0f90c4c24b66026048b5852ec (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=83830de54936404aa798ca5e4773319d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.104979 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df13afe0f90c4c24b66026048b5852ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.106081 9473 tablet_service.cc:1467] Processing CreateTablet for tablet b93fe374d2a9496a847480b7824a5937 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=83830de54936404aa798ca5e4773319d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.106395 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b93fe374d2a9496a847480b7824a5937. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.106938 7256 tablet_service.cc:1467] Processing CreateTablet for tablet f454ed0b1a924bec9d08f8740137be9b (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=83830de54936404aa798ca5e4773319d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.107255 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f454ed0b1a924bec9d08f8740137be9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.110275 9737 tablet_service.cc:1467] Processing CreateTablet for tablet b93fe374d2a9496a847480b7824a5937 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=83830de54936404aa798ca5e4773319d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.110620 9737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b93fe374d2a9496a847480b7824a5937. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.110819 13503 tablet_bootstrap.cc:492] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:47.112030 13503 tablet_bootstrap.cc:654] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.114104 13503 tablet_bootstrap.cc:492] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:47.114163 13502 tablet_bootstrap.cc:492] T df13afe0f90c4c24b66026048b5852ec P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:47.114264 13503 ts_tablet_manager.cc:1367] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:47.115129 13503 raft_consensus.cc:348] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:47.115304 13503 raft_consensus.cc:374] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.115341 13503 raft_consensus.cc:729] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.116528 13502 tablet_bootstrap.cc:654] T df13afe0f90c4c24b66026048b5852ec P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.115645 13503 consensus_queue.cc:260] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:47.117560 13503 ts_tablet_manager.cc:1398] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:59:47.120421 13502 tablet_bootstrap.cc:492] T df13afe0f90c4c24b66026048b5852ec P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:47.120618 13502 ts_tablet_manager.cc:1367] T df13afe0f90c4c24b66026048b5852ec P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 01:59:47.121309 13502 raft_consensus.cc:348] T df13afe0f90c4c24b66026048b5852ec P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:47.121762 13502 raft_consensus.cc:374] T df13afe0f90c4c24b66026048b5852ec P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.121827 13502 raft_consensus.cc:729] T df13afe0f90c4c24b66026048b5852ec P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.121975 13502 consensus_queue.cc:260] T df13afe0f90c4c24b66026048b5852ec P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:47.122282 13502 ts_tablet_manager.cc:1398] T df13afe0f90c4c24b66026048b5852ec P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:47.126387 9604 tablet_service.cc:1467] Processing CreateTablet for tablet b93fe374d2a9496a847480b7824a5937 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=83830de54936404aa798ca5e4773319d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.126742 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b93fe374d2a9496a847480b7824a5937. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.128166 6736 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:47.128309 6736 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:47.129978 13502 tablet_bootstrap.cc:492] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:47.131004 13502 tablet_bootstrap.cc:654] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.131842 6736 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 06e485b245f344e5a820973a86ae89a4
I20240503 01:59:47.132548 9341 tablet_service.cc:1514] Processing DeleteTablet for tablet 06e485b245f344e5a820973a86ae89a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:47 UTC) from {username='slave'} at 127.0.0.1:36040
I20240503 01:59:47.133139 9737 tablet_service.cc:1514] Processing DeleteTablet for tablet 06e485b245f344e5a820973a86ae89a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:47 UTC) from {username='slave'} at 127.0.0.1:48152
I20240503 01:59:47.134042 9472 tablet_service.cc:1514] Processing DeleteTablet for tablet 06e485b245f344e5a820973a86ae89a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:47 UTC) from {username='slave'} at 127.0.0.1:54678
I20240503 01:59:47.134399 13459 tablet_replica.cc:330] T 06e485b245f344e5a820973a86ae89a4 P bb5364d730cb4489a0bfc6c270b7450a: stopping tablet replica
I20240503 01:59:47.134608 13459 raft_consensus.cc:2229] T 06e485b245f344e5a820973a86ae89a4 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:47.134747 13459 raft_consensus.cc:2258] T 06e485b245f344e5a820973a86ae89a4 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:47.135190 9342 tablet_service.cc:1467] Processing CreateTablet for tablet f454ed0b1a924bec9d08f8740137be9b (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=83830de54936404aa798ca5e4773319d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.135442 13459 ts_tablet_manager.cc:1875] T 06e485b245f344e5a820973a86ae89a4 P bb5364d730cb4489a0bfc6c270b7450a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:47.135550 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f454ed0b1a924bec9d08f8740137be9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.137488 13459 ts_tablet_manager.cc:1888] T 06e485b245f344e5a820973a86ae89a4 P bb5364d730cb4489a0bfc6c270b7450a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:47.137575 13459 log.cc:1196] T 06e485b245f344e5a820973a86ae89a4 P bb5364d730cb4489a0bfc6c270b7450a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/06e485b245f344e5a820973a86ae89a4
I20240503 01:59:47.137835 13459 ts_tablet_manager.cc:1909] T 06e485b245f344e5a820973a86ae89a4 P bb5364d730cb4489a0bfc6c270b7450a: Deleting consensus metadata
I20240503 01:59:47.138020 13511 tablet_replica.cc:330] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8: stopping tablet replica
I20240503 01:59:47.138206 13511 raft_consensus.cc:2229] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:47.138484 13511 raft_consensus.cc:2258] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:47.138845 6725 catalog_manager.cc:4830] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): tablet 06e485b245f344e5a820973a86ae89a4 (table rebalancer_dynamic_table_new_name [id=6c45beac4ff3468c9ae1763f764ad167]) successfully deleted
I20240503 01:59:47.139278 13506 tablet_bootstrap.cc:492] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:47.139750 13511 ts_tablet_manager.cc:1875] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:47.143949 13505 tablet_bootstrap.cc:492] T f454ed0b1a924bec9d08f8740137be9b P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:47.144330 13509 tablet_bootstrap.cc:492] T df13afe0f90c4c24b66026048b5852ec P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:47.144619 13511 ts_tablet_manager.cc:1888] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:47.144711 13511 log.cc:1196] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/06e485b245f344e5a820973a86ae89a4
I20240503 01:59:47.145068 13511 ts_tablet_manager.cc:1909] T 06e485b245f344e5a820973a86ae89a4 P 12e8c683db054df3b75c6285b98a5ad8: Deleting consensus metadata
I20240503 01:59:47.145650 13509 tablet_bootstrap.cc:654] T df13afe0f90c4c24b66026048b5852ec P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.146317 13506 tablet_bootstrap.cc:654] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.147106 13509 tablet_bootstrap.cc:492] T df13afe0f90c4c24b66026048b5852ec P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:47.147218 13509 ts_tablet_manager.cc:1367] T df13afe0f90c4c24b66026048b5852ec P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:47.147675 13509 raft_consensus.cc:348] T df13afe0f90c4c24b66026048b5852ec P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:47.147791 13509 raft_consensus.cc:374] T df13afe0f90c4c24b66026048b5852ec P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.147827 13509 raft_consensus.cc:729] T df13afe0f90c4c24b66026048b5852ec P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.147972 13509 consensus_queue.cc:260] T df13afe0f90c4c24b66026048b5852ec P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:47.148294 13509 ts_tablet_manager.cc:1398] T df13afe0f90c4c24b66026048b5852ec P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:47.148384 13504 tablet_bootstrap.cc:492] T b93fe374d2a9496a847480b7824a5937 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:47.148468 13509 tablet_bootstrap.cc:492] T f454ed0b1a924bec9d08f8740137be9b P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:47.149369 13509 tablet_bootstrap.cc:654] T f454ed0b1a924bec9d08f8740137be9b P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.150641 13504 tablet_bootstrap.cc:654] T b93fe374d2a9496a847480b7824a5937 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.149181 13505 tablet_bootstrap.cc:654] T f454ed0b1a924bec9d08f8740137be9b P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.154188 13506 tablet_bootstrap.cc:492] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:47.154639 13506 ts_tablet_manager.cc:1367] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20240503 01:59:47.155444 13506 raft_consensus.cc:348] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:47.155282 6727 catalog_manager.cc:4830] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): tablet 06e485b245f344e5a820973a86ae89a4 (table rebalancer_dynamic_table_new_name [id=6c45beac4ff3468c9ae1763f764ad167]) successfully deleted
I20240503 01:59:47.155720 13506 raft_consensus.cc:374] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.155838 13506 raft_consensus.cc:729] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.156059 13509 tablet_bootstrap.cc:492] T f454ed0b1a924bec9d08f8740137be9b P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:47.156067 13506 consensus_queue.cc:260] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:47.156163 13509 ts_tablet_manager.cc:1367] T f454ed0b1a924bec9d08f8740137be9b P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240503 01:59:47.156245 13512 tablet_replica.cc:330] T 06e485b245f344e5a820973a86ae89a4 P db1a0000aaa346f79c0aa4b6c5105191: stopping tablet replica
I20240503 01:59:47.156401 13512 raft_consensus.cc:2229] T 06e485b245f344e5a820973a86ae89a4 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:47.156526 13512 raft_consensus.cc:2258] T 06e485b245f344e5a820973a86ae89a4 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:47.156658 13509 raft_consensus.cc:348] T f454ed0b1a924bec9d08f8740137be9b P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:47.156767 13509 raft_consensus.cc:374] T f454ed0b1a924bec9d08f8740137be9b P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.156795 13509 raft_consensus.cc:729] T f454ed0b1a924bec9d08f8740137be9b P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.156914 13509 consensus_queue.cc:260] T f454ed0b1a924bec9d08f8740137be9b P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:47.157232 13509 ts_tablet_manager.cc:1398] T f454ed0b1a924bec9d08f8740137be9b P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:47.157743 13502 tablet_bootstrap.cc:492] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:47.157742 13512 ts_tablet_manager.cc:1875] T 06e485b245f344e5a820973a86ae89a4 P db1a0000aaa346f79c0aa4b6c5105191: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:47.157838 13502 ts_tablet_manager.cc:1367] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.000s
I20240503 01:59:47.158353 13502 raft_consensus.cc:348] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:47.158474 13502 raft_consensus.cc:374] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.158504 13502 raft_consensus.cc:729] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.158668 13502 consensus_queue.cc:260] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:47.159029 13502 ts_tablet_manager.cc:1398] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:47.159355 13506 ts_tablet_manager.cc:1398] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:47.160560 13505 tablet_bootstrap.cc:492] T f454ed0b1a924bec9d08f8740137be9b P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:47.161387 13504 tablet_bootstrap.cc:492] T b93fe374d2a9496a847480b7824a5937 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:47.161554 13504 ts_tablet_manager.cc:1367] T b93fe374d2a9496a847480b7824a5937 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20240503 01:59:47.162091 13504 raft_consensus.cc:348] T b93fe374d2a9496a847480b7824a5937 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:47.162233 13504 raft_consensus.cc:374] T b93fe374d2a9496a847480b7824a5937 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.162269 13504 raft_consensus.cc:729] T b93fe374d2a9496a847480b7824a5937 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.162417 13504 consensus_queue.cc:260] T b93fe374d2a9496a847480b7824a5937 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:47.163461 13504 ts_tablet_manager.cc:1398] T b93fe374d2a9496a847480b7824a5937 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:47.163668 13504 tablet_bootstrap.cc:492] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:47.164815 13504 tablet_bootstrap.cc:654] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.166146 13504 tablet_bootstrap.cc:492] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:47.166981 13504 ts_tablet_manager.cc:1367] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:59:47.167325 13512 ts_tablet_manager.cc:1888] T 06e485b245f344e5a820973a86ae89a4 P db1a0000aaa346f79c0aa4b6c5105191: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:47.167423 13512 log.cc:1196] T 06e485b245f344e5a820973a86ae89a4 P db1a0000aaa346f79c0aa4b6c5105191: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/06e485b245f344e5a820973a86ae89a4
I20240503 01:59:47.167773 13512 ts_tablet_manager.cc:1909] T 06e485b245f344e5a820973a86ae89a4 P db1a0000aaa346f79c0aa4b6c5105191: Deleting consensus metadata
I20240503 01:59:47.167733 13504 raft_consensus.cc:348] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:47.167989 13504 raft_consensus.cc:374] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.168067 13504 raft_consensus.cc:729] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.168267 13504 consensus_queue.cc:260] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:47.169116 6726 catalog_manager.cc:4830] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): tablet 06e485b245f344e5a820973a86ae89a4 (table rebalancer_dynamic_table_new_name [id=6c45beac4ff3468c9ae1763f764ad167]) successfully deleted
I20240503 01:59:47.169396 13504 ts_tablet_manager.cc:1398] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:47.169716 13505 ts_tablet_manager.cc:1367] T f454ed0b1a924bec9d08f8740137be9b P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20240503 01:59:47.170464 13505 raft_consensus.cc:348] T f454ed0b1a924bec9d08f8740137be9b P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:47.171319 13505 raft_consensus.cc:374] T f454ed0b1a924bec9d08f8740137be9b P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.171428 13505 raft_consensus.cc:729] T f454ed0b1a924bec9d08f8740137be9b P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.171665 13505 consensus_queue.cc:260] T f454ed0b1a924bec9d08f8740137be9b P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:47.172083 13505 ts_tablet_manager.cc:1398] T f454ed0b1a924bec9d08f8740137be9b P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:47.187147 12070 raft_consensus.cc:482] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:47.187317 12070 raft_consensus.cc:504] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:47.187712 12070 leader_election.cc:288] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:47.188553 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df13afe0f90c4c24b66026048b5852ec" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:47.188596 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df13afe0f90c4c24b66026048b5852ec" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:47.188725 9363 raft_consensus.cc:2454] T df13afe0f90c4c24b66026048b5852ec P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:47.188746 9625 raft_consensus.cc:2454] T df13afe0f90c4c24b66026048b5852ec P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:47.189049 9429 leader_election.cc:302] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:47.189733 12070 raft_consensus.cc:2781] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:47.189795 12070 raft_consensus.cc:482] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:47.189831 12070 raft_consensus.cc:3037] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:47.191043 12070 raft_consensus.cc:504] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:47.191453 12070 leader_election.cc:288] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:47.191848 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df13afe0f90c4c24b66026048b5852ec" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:47.192003 9625 raft_consensus.cc:3037] T df13afe0f90c4c24b66026048b5852ec P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:47.192198 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df13afe0f90c4c24b66026048b5852ec" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:47.192318 9363 raft_consensus.cc:3037] T df13afe0f90c4c24b66026048b5852ec P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:47.193110 9625 raft_consensus.cc:2454] T df13afe0f90c4c24b66026048b5852ec P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:47.193385 9363 raft_consensus.cc:2454] T df13afe0f90c4c24b66026048b5852ec P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:47.193684 9429 leader_election.cc:302] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:47.193908 12070 raft_consensus.cc:2781] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:47.194006 12070 raft_consensus.cc:686] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Becoming Leader. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Running, Role: LEADER
I20240503 01:59:47.194330 12070 consensus_queue.cc:237] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:47.199337 6733 catalog_manager.cc:5484] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: term changed from 0 to 1, leader changed from <none> to bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:47.218295 9342 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:47.226321 7256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:47.232502 7124 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:47.233495 9604 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:47.233880 6861 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:47.235874 6993 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:47.240257 6737 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:47.246687 9472 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:47.268016 9737 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:47.272265 7124 tablet_service.cc:1467] Processing CreateTablet for tablet 8e3f5ade863a46739958cec03f16e1cc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c0446fb3704547f4a3600450dff3e2fd]), partition=
I20240503 01:59:47.272265 9604 tablet_service.cc:1467] Processing CreateTablet for tablet 8e3f5ade863a46739958cec03f16e1cc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c0446fb3704547f4a3600450dff3e2fd]), partition=
I20240503 01:59:47.273211 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e3f5ade863a46739958cec03f16e1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.273211 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e3f5ade863a46739958cec03f16e1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.278790 13349 raft_consensus.cc:482] T f454ed0b1a924bec9d08f8740137be9b P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:47.278987 13349 raft_consensus.cc:504] T f454ed0b1a924bec9d08f8740137be9b P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:47.279735 13349 leader_election.cc:288] T f454ed0b1a924bec9d08f8740137be9b P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:47.280897 9472 tablet_service.cc:1467] Processing CreateTablet for tablet 8e3f5ade863a46739958cec03f16e1cc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c0446fb3704547f4a3600450dff3e2fd]), partition=
I20240503 01:59:47.281265 9472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e3f5ade863a46739958cec03f16e1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.284360 13504 tablet_bootstrap.cc:492] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:47.285387 13504 tablet_bootstrap.cc:654] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.286798 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f454ed0b1a924bec9d08f8740137be9b" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:47.286972 7274 raft_consensus.cc:2454] T f454ed0b1a924bec9d08f8740137be9b P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:47.287410 9297 leader_election.cc:302] T f454ed0b1a924bec9d08f8740137be9b P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 1d0e738c34af40fa9950c6f086619b3a; no voters:
I20240503 01:59:47.287667 13540 tablet_bootstrap.cc:492] T 8e3f5ade863a46739958cec03f16e1cc P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:47.287721 13349 raft_consensus.cc:2781] T f454ed0b1a924bec9d08f8740137be9b P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:47.287793 13349 raft_consensus.cc:482] T f454ed0b1a924bec9d08f8740137be9b P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:47.287846 13349 raft_consensus.cc:3037] T f454ed0b1a924bec9d08f8740137be9b P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:47.288884 13540 tablet_bootstrap.cc:654] T 8e3f5ade863a46739958cec03f16e1cc P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.288959 13349 raft_consensus.cc:504] T f454ed0b1a924bec9d08f8740137be9b P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:47.289484 13349 leader_election.cc:288] T f454ed0b1a924bec9d08f8740137be9b P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:47.290026 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f454ed0b1a924bec9d08f8740137be9b" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:47.290174 7274 raft_consensus.cc:3037] T f454ed0b1a924bec9d08f8740137be9b P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:47.291263 7274 raft_consensus.cc:2454] T f454ed0b1a924bec9d08f8740137be9b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:47.292840 9297 leader_election.cc:302] T f454ed0b1a924bec9d08f8740137be9b P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 1d0e738c34af40fa9950c6f086619b3a; no voters:
I20240503 01:59:47.294817 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f454ed0b1a924bec9d08f8740137be9b" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:47.295338 6882 raft_consensus.cc:2454] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:47.296183 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f454ed0b1a924bec9d08f8740137be9b" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:47.296557 6881 raft_consensus.cc:3037] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:47.298650 13504 tablet_bootstrap.cc:492] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:47.298957 13504 ts_tablet_manager.cc:1367] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.002s
I20240503 01:59:47.298966 13457 raft_consensus.cc:2781] T f454ed0b1a924bec9d08f8740137be9b P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:47.300339 13457 raft_consensus.cc:686] T f454ed0b1a924bec9d08f8740137be9b P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Running, Role: LEADER
I20240503 01:59:47.300796 13457 consensus_queue.cc:237] T f454ed0b1a924bec9d08f8740137be9b P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:47.301718 6881 raft_consensus.cc:2454] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:47.304076 13504 raft_consensus.cc:348] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:47.304404 6737 catalog_manager.cc:5484] T f454ed0b1a924bec9d08f8740137be9b P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:47.311517 13502 tablet_bootstrap.cc:492] T 8e3f5ade863a46739958cec03f16e1cc P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:47.312682 13502 tablet_bootstrap.cc:654] T 8e3f5ade863a46739958cec03f16e1cc P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.322448 13502 tablet_bootstrap.cc:492] T 8e3f5ade863a46739958cec03f16e1cc P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:47.322614 13502 ts_tablet_manager.cc:1367] T 8e3f5ade863a46739958cec03f16e1cc P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240503 01:59:47.322629 13504 raft_consensus.cc:374] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.322762 13504 raft_consensus.cc:729] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.323027 13504 consensus_queue.cc:260] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:47.323173 13502 raft_consensus.cc:348] T 8e3f5ade863a46739958cec03f16e1cc P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:47.323325 13502 raft_consensus.cc:374] T 8e3f5ade863a46739958cec03f16e1cc P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.323357 13502 raft_consensus.cc:729] T 8e3f5ade863a46739958cec03f16e1cc P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.324438 13502 consensus_queue.cc:260] T 8e3f5ade863a46739958cec03f16e1cc P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:47.324753 13504 ts_tablet_manager.cc:1398] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240503 01:59:47.328843 13502 ts_tablet_manager.cc:1398] T 8e3f5ade863a46739958cec03f16e1cc P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 01:59:47.352308 13540 tablet_bootstrap.cc:492] T 8e3f5ade863a46739958cec03f16e1cc P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:47.352439 13540 ts_tablet_manager.cc:1367] T 8e3f5ade863a46739958cec03f16e1cc P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.065s user 0.002s sys 0.000s
I20240503 01:59:47.353271 13540 raft_consensus.cc:348] T 8e3f5ade863a46739958cec03f16e1cc P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:47.353447 13540 raft_consensus.cc:374] T 8e3f5ade863a46739958cec03f16e1cc P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.353824 13540 raft_consensus.cc:729] T 8e3f5ade863a46739958cec03f16e1cc P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.354112 13540 consensus_queue.cc:260] T 8e3f5ade863a46739958cec03f16e1cc P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:47.354429 13540 ts_tablet_manager.cc:1398] T 8e3f5ade863a46739958cec03f16e1cc P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:47.361747 13101 raft_consensus.cc:482] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:47.361958 13101 raft_consensus.cc:504] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:47.362689 13101 leader_election.cc:288] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:47.363570 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e3f5ade863a46739958cec03f16e1cc" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:47.363763 9625 raft_consensus.cc:2454] T 8e3f5ade863a46739958cec03f16e1cc P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:47.363858 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e3f5ade863a46739958cec03f16e1cc" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:47.363991 7145 raft_consensus.cc:2454] T 8e3f5ade863a46739958cec03f16e1cc P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:47.364446 9429 leader_election.cc:302] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:47.364634 13101 raft_consensus.cc:2781] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:47.364701 13101 raft_consensus.cc:482] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:47.364745 13101 raft_consensus.cc:3037] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:47.365720 13101 raft_consensus.cc:504] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:47.366101 13101 leader_election.cc:288] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:47.366479 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e3f5ade863a46739958cec03f16e1cc" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:47.366627 7145 raft_consensus.cc:3037] T 8e3f5ade863a46739958cec03f16e1cc P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:47.367470 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e3f5ade863a46739958cec03f16e1cc" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:47.367600 9625 raft_consensus.cc:3037] T 8e3f5ade863a46739958cec03f16e1cc P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:47.367780 7145 raft_consensus.cc:2454] T 8e3f5ade863a46739958cec03f16e1cc P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:47.368115 9429 leader_election.cc:302] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:47.368593 9625 raft_consensus.cc:2454] T 8e3f5ade863a46739958cec03f16e1cc P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:47.370493 13101 raft_consensus.cc:2781] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:47.370703 13101 raft_consensus.cc:686] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Becoming Leader. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Running, Role: LEADER
I20240503 01:59:47.370906 13101 consensus_queue.cc:237] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:47.375313 6733 catalog_manager.cc:5484] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: term changed from 0 to 1, leader changed from <none> to bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:47.379940 11786 raft_consensus.cc:482] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:47.380079 11786 raft_consensus.cc:504] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:47.380587 11786 leader_election.cc:288] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:47.380971 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b93fe374d2a9496a847480b7824a5937" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:47.381125 9755 raft_consensus.cc:2454] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 01:59:47.381217 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b93fe374d2a9496a847480b7824a5937" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:47.381361 9493 raft_consensus.cc:2454] T b93fe374d2a9496a847480b7824a5937 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 01:59:47.381729 9560 leader_election.cc:302] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:47.382004 11786 raft_consensus.cc:2781] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:47.382083 11786 raft_consensus.cc:482] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:47.382128 11786 raft_consensus.cc:3037] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:47.383097 11786 raft_consensus.cc:504] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:47.383700 11786 leader_election.cc:288] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Requested vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:47.384155 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b93fe374d2a9496a847480b7824a5937" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:47.384232 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b93fe374d2a9496a847480b7824a5937" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:47.384300 9755 raft_consensus.cc:3037] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:47.384344 9493 raft_consensus.cc:3037] T b93fe374d2a9496a847480b7824a5937 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:47.385488 9755 raft_consensus.cc:2454] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 01:59:47.385507 9493 raft_consensus.cc:2454] T b93fe374d2a9496a847480b7824a5937 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 01:59:47.385985 9560 leader_election.cc:302] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:47.386600 11786 raft_consensus.cc:2781] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:47.386713 11786 raft_consensus.cc:686] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 LEADER]: Becoming Leader. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Running, Role: LEADER
I20240503 01:59:47.386873 11786 consensus_queue.cc:237] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:47.392011 6733 catalog_manager.cc:5484] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:47.410102 7145 consensus_queue.cc:237] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:47.411893 7274 raft_consensus.cc:1261] T 79efb54e1f8143be868e9ef37f5f925d P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:47.412153 9755 raft_consensus.cc:1261] T 79efb54e1f8143be868e9ef37f5f925d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:47.412832 12031 consensus_queue.cc:1035] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:47.413081 12946 consensus_queue.cc:1035] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:47.417546 12946 raft_consensus.cc:2932] T 79efb54e1f8143be868e9ef37f5f925d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:47.418046 7274 raft_consensus.cc:2932] T 79efb54e1f8143be868e9ef37f5f925d P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:47.418056 9755 raft_consensus.cc:2932] T 79efb54e1f8143be868e9ef37f5f925d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:47.422878 6733 catalog_manager.cc:5484] T 79efb54e1f8143be868e9ef37f5f925d P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:47.430485 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet 79efb54e1f8143be868e9ef37f5f925d with delete_type TABLET_DATA_TOMBSTONED (TS 5dcc33cc621a4cbbae155f3230fb7c46 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:47.431517 13565 tablet_replica.cc:330] T 79efb54e1f8143be868e9ef37f5f925d P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:47.431875 13565 raft_consensus.cc:2229] T 79efb54e1f8143be868e9ef37f5f925d P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:47.432063 13565 raft_consensus.cc:2258] T 79efb54e1f8143be868e9ef37f5f925d P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:47.433594 13565 ts_tablet_manager.cc:1875] T 79efb54e1f8143be868e9ef37f5f925d P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:47.437762 13565 ts_tablet_manager.cc:1888] T 79efb54e1f8143be868e9ef37f5f925d P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 01:59:47.437863 13565 log.cc:1196] T 79efb54e1f8143be868e9ef37f5f925d P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/79efb54e1f8143be868e9ef37f5f925d
I20240503 01:59:47.438549 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 79efb54e1f8143be868e9ef37f5f925d (table rebalance_test_table_13 [id=06feff591c384a50825820a47b49c8f9]) successfully deleted
I20240503 01:59:47.489073 6737 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:47.495321 9625 raft_consensus.cc:1261] T 8e3f5ade863a46739958cec03f16e1cc P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:47.496269 7145 raft_consensus.cc:1261] T 8e3f5ade863a46739958cec03f16e1cc P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:47.497217 12070 consensus_queue.cc:1035] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:47.499151 12070 consensus_queue.cc:1035] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:47.504195 6737 catalog_manager.cc:5576] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087) does not have the latest schema for tablet 8e3f5ade863a46739958cec03f16e1cc (table rebalancer_dynamic_table [id=c0446fb3704547f4a3600450dff3e2fd]). Expected version 1 got 0
I20240503 01:59:47.506783 6733 catalog_manager.cc:5576] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979) does not have the latest schema for tablet 8e3f5ade863a46739958cec03f16e1cc (table rebalancer_dynamic_table [id=c0446fb3704547f4a3600450dff3e2fd]). Expected version 1 got 0
I20240503 01:59:47.507117 13401 tablet.cc:1722] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:47.507469 13401 tablet.cc:1620] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:47.511185 6740 catalog_manager.cc:5576] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087) does not have the latest schema for tablet 8e3f5ade863a46739958cec03f16e1cc (table rebalancer_dynamic_table [id=c0446fb3704547f4a3600450dff3e2fd]). Expected version 1 got 0
I20240503 01:59:47.515583 6739 catalog_manager.cc:5576] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979) does not have the latest schema for tablet 8e3f5ade863a46739958cec03f16e1cc (table rebalancer_dynamic_table [id=c0446fb3704547f4a3600450dff3e2fd]). Expected version 1 got 0
I20240503 01:59:47.516710 13610 tablet.cc:1722] T 8e3f5ade863a46739958cec03f16e1cc P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:47.517130 13610 tablet.cc:1620] T 8e3f5ade863a46739958cec03f16e1cc P b7062fbbc2b54fff9e99ec4ea5562c76: MemRowSet was empty: no flush needed.
I20240503 01:59:47.517875 13608 mvcc.cc:205] Tried to move back new op lower bound from 7023417702374383616 to 7023417701873795072. Current Snapshot: MvccSnapshot[applied={T|T < 7023417702374383616}]
I20240503 01:59:47.520231 13611 tablet.cc:1722] T 8e3f5ade863a46739958cec03f16e1cc P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:47.521075 13611 tablet.cc:1620] T 8e3f5ade863a46739958cec03f16e1cc P c688f3df56ac4b6d9b0c30fa4cc8ca2a: MemRowSet was empty: no flush needed.
I20240503 01:59:47.522428 6739 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=c0446fb3704547f4a3600450dff3e2fd] alter complete (version 1)
I20240503 01:59:47.592577 9604 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:47.595518 6861 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:47.603741 9737 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:47.605755 7256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:47.611204 9472 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:47.614755 7124 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:47.616092 6993 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:47.619417 9342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:47.629626 6741 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:47.639247 13401 tablet.cc:1722] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:47.639608 13401 tablet.cc:1620] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:47.640210 13610 tablet.cc:1722] T 8e3f5ade863a46739958cec03f16e1cc P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:47.640508 13610 tablet.cc:1620] T 8e3f5ade863a46739958cec03f16e1cc P b7062fbbc2b54fff9e99ec4ea5562c76: MemRowSet was empty: no flush needed.
I20240503 01:59:47.640782 13611 tablet.cc:1722] T 8e3f5ade863a46739958cec03f16e1cc P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:47.641037 13611 tablet.cc:1620] T 8e3f5ade863a46739958cec03f16e1cc P c688f3df56ac4b6d9b0c30fa4cc8ca2a: MemRowSet was empty: no flush needed.
I20240503 01:59:47.648977 6733 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=c0446fb3704547f4a3600450dff3e2fd] alter complete (version 2)
I20240503 01:59:47.694073 12070 consensus_queue.cc:1035] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:47.699057 13101 consensus_queue.cc:1035] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:47.718274 7014 consensus_queue.cc:237] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:47.720280 7274 raft_consensus.cc:1261] T 4be54233e31248cd8deafa6b7845a3f8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:47.720278 9757 raft_consensus.cc:1261] T 4be54233e31248cd8deafa6b7845a3f8 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:47.720927 13635 consensus_queue.cc:1035] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:47.721276 13636 consensus_queue.cc:1035] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:47.723814 12321 raft_consensus.cc:2932] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:47.724362 7274 raft_consensus.cc:2932] T 4be54233e31248cd8deafa6b7845a3f8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:47.724531 9757 raft_consensus.cc:2932] T 4be54233e31248cd8deafa6b7845a3f8 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:47.727762 6736 catalog_manager.cc:5484] T 4be54233e31248cd8deafa6b7845a3f8 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:47.730950 7124 tablet_service.cc:1514] Processing DeleteTablet for tablet 4be54233e31248cd8deafa6b7845a3f8 with delete_type TABLET_DATA_TOMBSTONED (TS c688f3df56ac4b6d9b0c30fa4cc8ca2a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:47.731550 13663 tablet_replica.cc:330] T 4be54233e31248cd8deafa6b7845a3f8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:47.731789 13663 raft_consensus.cc:2229] T 4be54233e31248cd8deafa6b7845a3f8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:47.732015 13663 raft_consensus.cc:2258] T 4be54233e31248cd8deafa6b7845a3f8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:47.732700 13663 ts_tablet_manager.cc:1875] T 4be54233e31248cd8deafa6b7845a3f8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:47.736579 13663 ts_tablet_manager.cc:1888] T 4be54233e31248cd8deafa6b7845a3f8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 01:59:47.736907 13663 log.cc:1196] T 4be54233e31248cd8deafa6b7845a3f8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/4be54233e31248cd8deafa6b7845a3f8
I20240503 01:59:47.737884 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet 4be54233e31248cd8deafa6b7845a3f8 (table rebalance_test_table_19 [id=d2d157b43e4243979ed58dfe4b2fe341]) successfully deleted
I20240503 01:59:47.751027 6733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:47.763641 13401 tablet.cc:1722] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:47.764117 13611 tablet.cc:1722] T 8e3f5ade863a46739958cec03f16e1cc P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:47.764657 13610 tablet.cc:1722] T 8e3f5ade863a46739958cec03f16e1cc P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:47.770709 6741 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=c0446fb3704547f4a3600450dff3e2fd] alter complete (version 3)
I20240503 01:59:47.774329 13010 consensus_queue.cc:1035] T f454ed0b1a924bec9d08f8740137be9b P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:47.781435 13349 consensus_queue.cc:1035] T f454ed0b1a924bec9d08f8740137be9b P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:47.801116 11786 consensus_queue.cc:1035] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:47.811669 13434 consensus_queue.cc:1035] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:47.862047 9342 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:47.865437 6861 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:47.869750 7256 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:47.876039 6737 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:47.876273 6737 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:47.880442 6737 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 8e3f5ade863a46739958cec03f16e1cc
I20240503 01:59:47.881419 9604 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:47.881733 9605 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e3f5ade863a46739958cec03f16e1cc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:47 UTC) from {username='slave'} at 127.0.0.1:58562
I20240503 01:59:47.881767 7124 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e3f5ade863a46739958cec03f16e1cc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:47 UTC) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:47.882148 13663 tablet_replica.cc:330] T 8e3f5ade863a46739958cec03f16e1cc P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:47.882341 13663 raft_consensus.cc:2229] T 8e3f5ade863a46739958cec03f16e1cc P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:47.882571 13663 raft_consensus.cc:2258] T 8e3f5ade863a46739958cec03f16e1cc P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:47.884212 13663 ts_tablet_manager.cc:1875] T 8e3f5ade863a46739958cec03f16e1cc P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:47.886662 9472 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e3f5ade863a46739958cec03f16e1cc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:47 UTC) from {username='slave'} at 127.0.0.1:54678
I20240503 01:59:47.887677 9472 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:47.900404 13663 ts_tablet_manager.cc:1888] T 8e3f5ade863a46739958cec03f16e1cc P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:47.900525 13663 log.cc:1196] T 8e3f5ade863a46739958cec03f16e1cc P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/8e3f5ade863a46739958cec03f16e1cc
I20240503 01:59:47.901108 13663 ts_tablet_manager.cc:1909] T 8e3f5ade863a46739958cec03f16e1cc P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting consensus metadata
I20240503 01:59:47.903666 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet 8e3f5ade863a46739958cec03f16e1cc (table rebalancer_dynamic_table_new_name [id=c0446fb3704547f4a3600450dff3e2fd]) successfully deleted
I20240503 01:59:47.903846 7124 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:47.911268 13721 tablet_replica.cc:330] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a: stopping tablet replica
I20240503 01:59:47.916267 13721 raft_consensus.cc:2229] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:47.916929 13721 raft_consensus.cc:2258] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:47.918143 13721 ts_tablet_manager.cc:1875] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:47.918215 6993 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:47.921411 13721 ts_tablet_manager.cc:1888] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:47.921602 13721 log.cc:1196] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/8e3f5ade863a46739958cec03f16e1cc
I20240503 01:59:47.921976 13721 ts_tablet_manager.cc:1909] T 8e3f5ade863a46739958cec03f16e1cc P bb5364d730cb4489a0bfc6c270b7450a: Deleting consensus metadata
I20240503 01:59:47.926885 9737 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:47.929728 6725 catalog_manager.cc:4830] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): tablet 8e3f5ade863a46739958cec03f16e1cc (table rebalancer_dynamic_table_new_name [id=c0446fb3704547f4a3600450dff3e2fd]) successfully deleted
I20240503 01:59:47.931129 6737 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:47.951174 9737 tablet_service.cc:1467] Processing CreateTablet for tablet 65b41c24a2504ad386e437c24f1decee (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=9a04da30c0934e269de5ed55fe118bee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.951237 9605 tablet_service.cc:1467] Processing CreateTablet for tablet 65b41c24a2504ad386e437c24f1decee (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=9a04da30c0934e269de5ed55fe118bee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.951551 9605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b41c24a2504ad386e437c24f1decee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.951558 9737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b41c24a2504ad386e437c24f1decee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.954102 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 22b6ec13c4e04d6cbade150f0c644e52 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=9a04da30c0934e269de5ed55fe118bee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.954658 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b6ec13c4e04d6cbade150f0c644e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.958339 6993 tablet_service.cc:1467] Processing CreateTablet for tablet 65b41c24a2504ad386e437c24f1decee (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=9a04da30c0934e269de5ed55fe118bee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.958701 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b41c24a2504ad386e437c24f1decee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.960793 7124 tablet_service.cc:1467] Processing CreateTablet for tablet 22b6ec13c4e04d6cbade150f0c644e52 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=9a04da30c0934e269de5ed55fe118bee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.961127 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b6ec13c4e04d6cbade150f0c644e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.961241 7256 tablet_service.cc:1467] Processing CreateTablet for tablet b5ccbd83016a460f8870ae8ad44b8bdf (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=9a04da30c0934e269de5ed55fe118bee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.961490 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5ccbd83016a460f8870ae8ad44b8bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.962713 9472 tablet_service.cc:1467] Processing CreateTablet for tablet 22b6ec13c4e04d6cbade150f0c644e52 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=9a04da30c0934e269de5ed55fe118bee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.963266 9472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b6ec13c4e04d6cbade150f0c644e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.963521 13720 tablet_replica.cc:330] T 8e3f5ade863a46739958cec03f16e1cc P b7062fbbc2b54fff9e99ec4ea5562c76: stopping tablet replica
I20240503 01:59:47.963562 6861 tablet_service.cc:1467] Processing CreateTablet for tablet b5ccbd83016a460f8870ae8ad44b8bdf (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=9a04da30c0934e269de5ed55fe118bee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.963726 13720 raft_consensus.cc:2229] T 8e3f5ade863a46739958cec03f16e1cc P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:47.963840 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5ccbd83016a460f8870ae8ad44b8bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.963886 13720 raft_consensus.cc:2258] T 8e3f5ade863a46739958cec03f16e1cc P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:47.966604 13725 tablet_bootstrap.cc:492] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:47.968254 13720 ts_tablet_manager.cc:1875] T 8e3f5ade863a46739958cec03f16e1cc P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:47.968446 13725 tablet_bootstrap.cc:654] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.969323 9472 tablet_service.cc:1467] Processing CreateTablet for tablet b5ccbd83016a460f8870ae8ad44b8bdf (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=9a04da30c0934e269de5ed55fe118bee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:47.969651 9472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5ccbd83016a460f8870ae8ad44b8bdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:47.975813 13727 tablet_bootstrap.cc:492] T b5ccbd83016a460f8870ae8ad44b8bdf P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:47.977128 13727 tablet_bootstrap.cc:654] T b5ccbd83016a460f8870ae8ad44b8bdf P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.978348 13729 tablet_bootstrap.cc:492] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:47.979130 13722 tablet_bootstrap.cc:492] T 65b41c24a2504ad386e437c24f1decee P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:47.980942 13722 tablet_bootstrap.cc:654] T 65b41c24a2504ad386e437c24f1decee P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.981776 13729 tablet_bootstrap.cc:654] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.983484 13725 tablet_bootstrap.cc:492] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:47.983585 13727 tablet_bootstrap.cc:492] T b5ccbd83016a460f8870ae8ad44b8bdf P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:47.983707 13727 ts_tablet_manager.cc:1367] T b5ccbd83016a460f8870ae8ad44b8bdf P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 01:59:47.983708 13725 ts_tablet_manager.cc:1367] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240503 01:59:47.984274 13727 raft_consensus.cc:348] T b5ccbd83016a460f8870ae8ad44b8bdf P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:47.984407 13727 raft_consensus.cc:374] T b5ccbd83016a460f8870ae8ad44b8bdf P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.984443 13727 raft_consensus.cc:729] T b5ccbd83016a460f8870ae8ad44b8bdf P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.984592 13727 consensus_queue.cc:260] T b5ccbd83016a460f8870ae8ad44b8bdf P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:47.985296 6737 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:47.985718 13727 ts_tablet_manager.cc:1398] T b5ccbd83016a460f8870ae8ad44b8bdf P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:47.986020 13720 ts_tablet_manager.cc:1888] T 8e3f5ade863a46739958cec03f16e1cc P b7062fbbc2b54fff9e99ec4ea5562c76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:47.986095 13720 log.cc:1196] T 8e3f5ade863a46739958cec03f16e1cc P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/8e3f5ade863a46739958cec03f16e1cc
I20240503 01:59:47.986367 13720 ts_tablet_manager.cc:1909] T 8e3f5ade863a46739958cec03f16e1cc P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting consensus metadata
I20240503 01:59:47.986811 13729 tablet_bootstrap.cc:492] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:47.986927 13729 ts_tablet_manager.cc:1367] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.001s
I20240503 01:59:47.987434 13729 raft_consensus.cc:348] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:47.987726 13729 raft_consensus.cc:374] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:47.987823 13729 raft_consensus.cc:729] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:47.988057 13729 consensus_queue.cc:260] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:47.988437 13729 ts_tablet_manager.cc:1398] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:47.991364 13726 tablet_bootstrap.cc:492] T 22b6ec13c4e04d6cbade150f0c644e52 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:47.993579 13726 tablet_bootstrap.cc:654] T 22b6ec13c4e04d6cbade150f0c644e52 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:47.994714 6727 catalog_manager.cc:4830] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): tablet 8e3f5ade863a46739958cec03f16e1cc (table rebalancer_dynamic_table_new_name [id=c0446fb3704547f4a3600450dff3e2fd]) successfully deleted
I20240503 01:59:47.995059 13724 tablet_bootstrap.cc:492] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:47.996737 13724 tablet_bootstrap.cc:654] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.001686 13725 raft_consensus.cc:348] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:48.001806 13730 tablet_bootstrap.cc:492] T 65b41c24a2504ad386e437c24f1decee P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:48.002004 13725 raft_consensus.cc:374] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.002120 13725 raft_consensus.cc:729] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.002368 13725 consensus_queue.cc:260] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:48.002954 13725 ts_tablet_manager.cc:1398] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:48.004334 13723 tablet_bootstrap.cc:492] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:48.004621 13722 tablet_bootstrap.cc:492] T 65b41c24a2504ad386e437c24f1decee P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:48.006738 13722 ts_tablet_manager.cc:1367] T 65b41c24a2504ad386e437c24f1decee P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20240503 01:59:48.006762 13730 tablet_bootstrap.cc:654] T 65b41c24a2504ad386e437c24f1decee P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.007540 13723 tablet_bootstrap.cc:654] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.007522 13722 raft_consensus.cc:348] T 65b41c24a2504ad386e437c24f1decee P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:48.007704 13722 raft_consensus.cc:374] T 65b41c24a2504ad386e437c24f1decee P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.007779 13722 raft_consensus.cc:729] T 65b41c24a2504ad386e437c24f1decee P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.007943 13722 consensus_queue.cc:260] T 65b41c24a2504ad386e437c24f1decee P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:48.008370 13722 ts_tablet_manager.cc:1398] T 65b41c24a2504ad386e437c24f1decee P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:48.009075 13730 tablet_bootstrap.cc:492] T 65b41c24a2504ad386e437c24f1decee P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:48.009970 13730 ts_tablet_manager.cc:1367] T 65b41c24a2504ad386e437c24f1decee P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240503 01:59:48.010695 13730 raft_consensus.cc:348] T 65b41c24a2504ad386e437c24f1decee P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:48.010831 13730 raft_consensus.cc:374] T 65b41c24a2504ad386e437c24f1decee P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.011401 13726 tablet_bootstrap.cc:492] T 22b6ec13c4e04d6cbade150f0c644e52 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:48.011526 13726 ts_tablet_manager.cc:1367] T 22b6ec13c4e04d6cbade150f0c644e52 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.004s
I20240503 01:59:48.012064 13726 raft_consensus.cc:348] T 22b6ec13c4e04d6cbade150f0c644e52 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:48.012727 13723 tablet_bootstrap.cc:492] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:48.013073 13723 ts_tablet_manager.cc:1367] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240503 01:59:48.013787 13723 raft_consensus.cc:348] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:48.014384 13724 tablet_bootstrap.cc:492] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:48.014542 13730 raft_consensus.cc:729] T 65b41c24a2504ad386e437c24f1decee P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.014802 13726 raft_consensus.cc:374] T 22b6ec13c4e04d6cbade150f0c644e52 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.014776 13730 consensus_queue.cc:260] T 65b41c24a2504ad386e437c24f1decee P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:48.015034 13726 raft_consensus.cc:729] T 22b6ec13c4e04d6cbade150f0c644e52 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.015162 13730 ts_tablet_manager.cc:1398] T 65b41c24a2504ad386e437c24f1decee P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 01:59:48.015331 13724 ts_tablet_manager.cc:1367] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240503 01:59:48.016000 13724 raft_consensus.cc:348] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:48.016310 13724 raft_consensus.cc:374] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.016405 13724 raft_consensus.cc:729] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.016595 13724 consensus_queue.cc:260] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:48.017241 13724 ts_tablet_manager.cc:1398] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:48.018110 13723 raft_consensus.cc:374] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.018190 13723 raft_consensus.cc:729] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.015333 13726 consensus_queue.cc:260] T 22b6ec13c4e04d6cbade150f0c644e52 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:48.018805 13723 consensus_queue.cc:260] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:48.019258 13726 ts_tablet_manager.cc:1398] T 22b6ec13c4e04d6cbade150f0c644e52 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 01:59:48.019626 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 75643ebd6e5f44e5bc3cbb452514f2ec (DEFAULT_TABLE table=rebalancer_dynamic_table [id=865eb68bf451427fa36c8409a20305a8]), partition=
I20240503 01:59:48.019934 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75643ebd6e5f44e5bc3cbb452514f2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.019963 13723 ts_tablet_manager.cc:1398] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 01:59:48.020198 13726 tablet_bootstrap.cc:492] T b5ccbd83016a460f8870ae8ad44b8bdf P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:48.021229 13726 tablet_bootstrap.cc:654] T b5ccbd83016a460f8870ae8ad44b8bdf P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.022568 13726 tablet_bootstrap.cc:492] T b5ccbd83016a460f8870ae8ad44b8bdf P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:48.022655 13726 ts_tablet_manager.cc:1367] T b5ccbd83016a460f8870ae8ad44b8bdf P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:48.023056 13726 raft_consensus.cc:348] T b5ccbd83016a460f8870ae8ad44b8bdf P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:48.023176 13726 raft_consensus.cc:374] T b5ccbd83016a460f8870ae8ad44b8bdf P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.023208 13726 raft_consensus.cc:729] T b5ccbd83016a460f8870ae8ad44b8bdf P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.023381 13726 consensus_queue.cc:260] T b5ccbd83016a460f8870ae8ad44b8bdf P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:48.023651 13726 ts_tablet_manager.cc:1398] T b5ccbd83016a460f8870ae8ad44b8bdf P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:48.023597 9737 tablet_service.cc:1467] Processing CreateTablet for tablet 75643ebd6e5f44e5bc3cbb452514f2ec (DEFAULT_TABLE table=rebalancer_dynamic_table [id=865eb68bf451427fa36c8409a20305a8]), partition=
I20240503 01:59:48.024085 9737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75643ebd6e5f44e5bc3cbb452514f2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.027997 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 75643ebd6e5f44e5bc3cbb452514f2ec (DEFAULT_TABLE table=rebalancer_dynamic_table [id=865eb68bf451427fa36c8409a20305a8]), partition=
I20240503 01:59:48.028532 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75643ebd6e5f44e5bc3cbb452514f2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.034554 13723 tablet_bootstrap.cc:492] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:48.036401 13727 tablet_bootstrap.cc:492] T 75643ebd6e5f44e5bc3cbb452514f2ec P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:48.036451 13730 tablet_bootstrap.cc:492] T 75643ebd6e5f44e5bc3cbb452514f2ec P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:48.036989 13723 tablet_bootstrap.cc:654] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.037829 13730 tablet_bootstrap.cc:654] T 75643ebd6e5f44e5bc3cbb452514f2ec P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.037954 13727 tablet_bootstrap.cc:654] T 75643ebd6e5f44e5bc3cbb452514f2ec P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.040205 13727 tablet_bootstrap.cc:492] T 75643ebd6e5f44e5bc3cbb452514f2ec P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:48.040274 13730 tablet_bootstrap.cc:492] T 75643ebd6e5f44e5bc3cbb452514f2ec P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:48.040335 13727 ts_tablet_manager.cc:1367] T 75643ebd6e5f44e5bc3cbb452514f2ec P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 01:59:48.040357 13730 ts_tablet_manager.cc:1367] T 75643ebd6e5f44e5bc3cbb452514f2ec P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 01:59:48.040849 13727 raft_consensus.cc:348] T 75643ebd6e5f44e5bc3cbb452514f2ec P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:48.040849 13730 raft_consensus.cc:348] T 75643ebd6e5f44e5bc3cbb452514f2ec P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:48.040977 13727 raft_consensus.cc:374] T 75643ebd6e5f44e5bc3cbb452514f2ec P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.040977 13730 raft_consensus.cc:374] T 75643ebd6e5f44e5bc3cbb452514f2ec P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.041011 13727 raft_consensus.cc:729] T 75643ebd6e5f44e5bc3cbb452514f2ec P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.041011 13730 raft_consensus.cc:729] T 75643ebd6e5f44e5bc3cbb452514f2ec P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.041150 13730 consensus_queue.cc:260] T 75643ebd6e5f44e5bc3cbb452514f2ec P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:48.041494 13730 ts_tablet_manager.cc:1398] T 75643ebd6e5f44e5bc3cbb452514f2ec P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:48.042680 13727 consensus_queue.cc:260] T 75643ebd6e5f44e5bc3cbb452514f2ec P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:48.046034 13727 ts_tablet_manager.cc:1398] T 75643ebd6e5f44e5bc3cbb452514f2ec P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 01:59:48.048537 12366 raft_consensus.cc:482] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:48.048698 12366 raft_consensus.cc:504] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:48.049124 12366 leader_election.cc:288] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:48.049847 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ccbd83016a460f8870ae8ad44b8bdf" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:48.050014 6881 raft_consensus.cc:2454] T b5ccbd83016a460f8870ae8ad44b8bdf P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:48.050580 7212 leader_election.cc:302] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:48.050515 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ccbd83016a460f8870ae8ad44b8bdf" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:48.050716 9494 raft_consensus.cc:2454] T b5ccbd83016a460f8870ae8ad44b8bdf P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:48.053655 12366 raft_consensus.cc:2781] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:48.053759 12366 raft_consensus.cc:482] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:48.053795 12366 raft_consensus.cc:3037] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.055653 12366 raft_consensus.cc:504] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:48.057027 13723 tablet_bootstrap.cc:492] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:48.057164 13723 ts_tablet_manager.cc:1367] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20240503 01:59:48.057376 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ccbd83016a460f8870ae8ad44b8bdf" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:48.057585 6881 raft_consensus.cc:3037] T b5ccbd83016a460f8870ae8ad44b8bdf P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.057746 13723 raft_consensus.cc:348] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:48.057854 13723 raft_consensus.cc:374] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.057883 13723 raft_consensus.cc:729] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.058008 13723 consensus_queue.cc:260] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:48.058323 13723 ts_tablet_manager.cc:1398] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:48.058990 6881 raft_consensus.cc:2454] T b5ccbd83016a460f8870ae8ad44b8bdf P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:48.060133 7212 leader_election.cc:302] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:48.060379 12963 raft_consensus.cc:2781] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:48.060599 12963 raft_consensus.cc:686] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 01:59:48.060752 12963 consensus_queue.cc:237] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:48.061398 12366 leader_election.cc:288] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Requested vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:48.061821 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ccbd83016a460f8870ae8ad44b8bdf" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:48.061961 9494 raft_consensus.cc:3037] T b5ccbd83016a460f8870ae8ad44b8bdf P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.063329 9494 raft_consensus.cc:2454] T b5ccbd83016a460f8870ae8ad44b8bdf P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:48.065848 6738 catalog_manager.cc:5484] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:48.083184 13349 raft_consensus.cc:482] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:48.083474 13349 raft_consensus.cc:504] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:48.084013 13349 leader_election.cc:288] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:48.084592 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b6ec13c4e04d6cbade150f0c644e52" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:48.084743 7145 raft_consensus.cc:2454] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:48.085119 9298 leader_election.cc:302] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:48.085350 13349 raft_consensus.cc:2781] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:48.085402 13349 raft_consensus.cc:482] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:48.085438 13349 raft_consensus.cc:3037] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.086727 13349 raft_consensus.cc:504] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:48.087155 13349 leader_election.cc:288] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Requested vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:48.087926 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b6ec13c4e04d6cbade150f0c644e52" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:48.088073 9493 raft_consensus.cc:3037] T 22b6ec13c4e04d6cbade150f0c644e52 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.089121 9493 raft_consensus.cc:2454] T 22b6ec13c4e04d6cbade150f0c644e52 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:48.089789 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b6ec13c4e04d6cbade150f0c644e52" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:48.089874 9296 leader_election.cc:302] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:48.089927 7145 raft_consensus.cc:3037] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.090150 13349 raft_consensus.cc:2781] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:48.090236 13349 raft_consensus.cc:686] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Running, Role: LEADER
I20240503 01:59:48.090376 13349 consensus_queue.cc:237] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:48.091070 7145 raft_consensus.cc:2454] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:48.092561 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b6ec13c4e04d6cbade150f0c644e52" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:48.092700 9494 raft_consensus.cc:2362] T 22b6ec13c4e04d6cbade150f0c644e52 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1. Re-sending same reply.
I20240503 01:59:48.096452 6738 catalog_manager.cc:5484] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:48.115119 7274 consensus_queue.cc:237] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:48.117372 7014 raft_consensus.cc:1261] T 9cc3435afe5144f9a52a286ac86d234f P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:48.117834 9494 raft_consensus.cc:1261] T 9cc3435afe5144f9a52a286ac86d234f P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:48.118806 12366 consensus_queue.cc:1035] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:48.119165 12366 consensus_queue.cc:1035] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:48.121629 12963 raft_consensus.cc:2932] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:48.123236 9494 raft_consensus.cc:2932] T 9cc3435afe5144f9a52a286ac86d234f P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:48.124634 7014 raft_consensus.cc:2932] T 9cc3435afe5144f9a52a286ac86d234f P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:48.130872 6735 catalog_manager.cc:5484] T 9cc3435afe5144f9a52a286ac86d234f P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:48.132978 13434 raft_consensus.cc:482] T 65b41c24a2504ad386e437c24f1decee P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:48.133136 13434 raft_consensus.cc:504] T 65b41c24a2504ad386e437c24f1decee P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:48.133648 13434 leader_election.cc:288] T 65b41c24a2504ad386e437c24f1decee P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:48.134096 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b41c24a2504ad386e437c24f1decee" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:48.134255 7012 raft_consensus.cc:2454] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 01:59:48.135560 9559 leader_election.cc:302] T 65b41c24a2504ad386e437c24f1decee P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:48.136173 9342 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cc3435afe5144f9a52a286ac86d234f with delete_type TABLET_DATA_TOMBSTONED (TS 12e8c683db054df3b75c6285b98a5ad8 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:36040
I20240503 01:59:48.136205 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b41c24a2504ad386e437c24f1decee" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:48.136354 9757 raft_consensus.cc:2454] T 65b41c24a2504ad386e437c24f1decee P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 01:59:48.137209 13764 tablet_replica.cc:330] T 9cc3435afe5144f9a52a286ac86d234f P 12e8c683db054df3b75c6285b98a5ad8: stopping tablet replica
I20240503 01:59:48.137223 13434 raft_consensus.cc:2781] T 65b41c24a2504ad386e437c24f1decee P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:48.137290 13434 raft_consensus.cc:482] T 65b41c24a2504ad386e437c24f1decee P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:48.137333 13434 raft_consensus.cc:3037] T 65b41c24a2504ad386e437c24f1decee P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.137532 13764 raft_consensus.cc:2229] T 9cc3435afe5144f9a52a286ac86d234f P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:48.137717 13764 raft_consensus.cc:2258] T 9cc3435afe5144f9a52a286ac86d234f P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:48.138226 13764 ts_tablet_manager.cc:1875] T 9cc3435afe5144f9a52a286ac86d234f P 12e8c683db054df3b75c6285b98a5ad8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:48.138614 13434 raft_consensus.cc:504] T 65b41c24a2504ad386e437c24f1decee P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:48.139061 13434 leader_election.cc:288] T 65b41c24a2504ad386e437c24f1decee P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Requested vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:48.139417 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b41c24a2504ad386e437c24f1decee" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:48.139425 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b41c24a2504ad386e437c24f1decee" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:48.139546 9757 raft_consensus.cc:3037] T 65b41c24a2504ad386e437c24f1decee P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.139560 7014 raft_consensus.cc:3037] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.140738 7014 raft_consensus.cc:2454] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 01:59:48.140671 9757 raft_consensus.cc:2454] T 65b41c24a2504ad386e437c24f1decee P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 01:59:48.141121 9559 leader_election.cc:302] T 65b41c24a2504ad386e437c24f1decee P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:48.141332 13434 raft_consensus.cc:2781] T 65b41c24a2504ad386e437c24f1decee P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:48.141445 13434 raft_consensus.cc:686] T 65b41c24a2504ad386e437c24f1decee P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 LEADER]: Becoming Leader. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Running, Role: LEADER
I20240503 01:59:48.141606 13434 consensus_queue.cc:237] T 65b41c24a2504ad386e437c24f1decee P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:48.144095 6737 catalog_manager.cc:5484] T 65b41c24a2504ad386e437c24f1decee P b7062fbbc2b54fff9e99ec4ea5562c76 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:48.147385 13764 ts_tablet_manager.cc:1888] T 9cc3435afe5144f9a52a286ac86d234f P 12e8c683db054df3b75c6285b98a5ad8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 01:59:48.147500 13764 log.cc:1196] T 9cc3435afe5144f9a52a286ac86d234f P 12e8c683db054df3b75c6285b98a5ad8: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/9cc3435afe5144f9a52a286ac86d234f
I20240503 01:59:48.148233 6727 catalog_manager.cc:4830] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): tablet 9cc3435afe5144f9a52a286ac86d234f (table rebalancer_extra_table_4 [id=5108cdf748db49d3858f16e7dd630a09]) successfully deleted
I20240503 01:59:48.195165 13349 raft_consensus.cc:482] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:48.195289 13349 raft_consensus.cc:504] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:48.195736 13349 leader_election.cc:288] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:48.196226 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75643ebd6e5f44e5bc3cbb452514f2ec" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:48.196377 6881 raft_consensus.cc:2454] T 75643ebd6e5f44e5bc3cbb452514f2ec P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:48.196288 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75643ebd6e5f44e5bc3cbb452514f2ec" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:48.196528 9757 raft_consensus.cc:2454] T 75643ebd6e5f44e5bc3cbb452514f2ec P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:48.197139 9299 leader_election.cc:302] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:48.197394 13349 raft_consensus.cc:2781] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:48.197465 13349 raft_consensus.cc:482] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:48.197546 13349 raft_consensus.cc:3037] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.198730 13349 raft_consensus.cc:504] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:48.199213 13349 leader_election.cc:288] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Requested vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:48.199755 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75643ebd6e5f44e5bc3cbb452514f2ec" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:48.199899 9757 raft_consensus.cc:3037] T 75643ebd6e5f44e5bc3cbb452514f2ec P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.200012 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75643ebd6e5f44e5bc3cbb452514f2ec" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:48.200155 6881 raft_consensus.cc:3037] T 75643ebd6e5f44e5bc3cbb452514f2ec P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.201049 9757 raft_consensus.cc:2454] T 75643ebd6e5f44e5bc3cbb452514f2ec P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:48.201469 6881 raft_consensus.cc:2454] T 75643ebd6e5f44e5bc3cbb452514f2ec P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:48.205060 9299 leader_election.cc:302] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:48.205410 13457 raft_consensus.cc:2781] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:48.205627 13457 raft_consensus.cc:686] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Running, Role: LEADER
I20240503 01:59:48.205799 13457 consensus_queue.cc:237] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:48.208827 6732 catalog_manager.cc:5484] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:48.267287 6993 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:48.276015 9605 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:48.278174 7256 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:48.278288 9342 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:48.281899 9737 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:48.282341 6861 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:48.301667 7124 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:48.331637 9472 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:48.346083 6735 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:48.355572 6881 raft_consensus.cc:1261] T 75643ebd6e5f44e5bc3cbb452514f2ec P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:48.356235 13457 consensus_queue.cc:1035] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:48.360095 6735 catalog_manager.cc:5576] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943) does not have the latest schema for tablet 75643ebd6e5f44e5bc3cbb452514f2ec (table rebalancer_dynamic_table [id=865eb68bf451427fa36c8409a20305a8]). Expected version 1 got 0
I20240503 01:59:48.362625 9757 raft_consensus.cc:1261] T 75643ebd6e5f44e5bc3cbb452514f2ec P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:48.363286 13457 consensus_queue.cc:1035] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:48.364681 13822 tablet.cc:1722] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:48.365381 13822 tablet.cc:1620] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 01:59:48.367184 6733 catalog_manager.cc:5576] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053) does not have the latest schema for tablet 75643ebd6e5f44e5bc3cbb452514f2ec (table rebalancer_dynamic_table [id=865eb68bf451427fa36c8409a20305a8]). Expected version 1 got 0
I20240503 01:59:48.369031 6735 catalog_manager.cc:5576] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943) does not have the latest schema for tablet 75643ebd6e5f44e5bc3cbb452514f2ec (table rebalancer_dynamic_table [id=865eb68bf451427fa36c8409a20305a8]). Expected version 1 got 0
I20240503 01:59:48.371742 13823 tablet.cc:1722] T 75643ebd6e5f44e5bc3cbb452514f2ec P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:48.372359 13823 tablet.cc:1620] T 75643ebd6e5f44e5bc3cbb452514f2ec P 5dcc33cc621a4cbbae155f3230fb7c46: MemRowSet was empty: no flush needed.
I20240503 01:59:48.379799 6735 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=865eb68bf451427fa36c8409a20305a8] alter complete (version 1)
I20240503 01:59:48.384608 13826 tablet.cc:1722] T 75643ebd6e5f44e5bc3cbb452514f2ec P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:48.384548 6735 catalog_manager.cc:5576] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053) does not have the latest schema for tablet 75643ebd6e5f44e5bc3cbb452514f2ec (table rebalancer_dynamic_table [id=865eb68bf451427fa36c8409a20305a8]). Expected version 1 got 0
I20240503 01:59:48.385078 13826 tablet.cc:1620] T 75643ebd6e5f44e5bc3cbb452514f2ec P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:48.411123 7143 consensus_queue.cc:237] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:48.412642 9625 raft_consensus.cc:1261] T dfdcdd54606e4320a5d0f506eb6dabf8 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:48.413144 7274 raft_consensus.cc:1261] T dfdcdd54606e4320a5d0f506eb6dabf8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:48.413543 12946 consensus_queue.cc:1035] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:48.414327 12946 consensus_queue.cc:1035] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:48.415853 12946 raft_consensus.cc:2932] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:48.416978 9625 raft_consensus.cc:2932] T dfdcdd54606e4320a5d0f506eb6dabf8 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:48.417930 7274 raft_consensus.cc:2932] T dfdcdd54606e4320a5d0f506eb6dabf8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:48.419553 6735 catalog_manager.cc:5484] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:48.426697 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet dfdcdd54606e4320a5d0f506eb6dabf8 with delete_type TABLET_DATA_TOMBSTONED (TS 5dcc33cc621a4cbbae155f3230fb7c46 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:48.428074 13850 tablet_replica.cc:330] T dfdcdd54606e4320a5d0f506eb6dabf8 P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:48.428256 13850 raft_consensus.cc:2229] T dfdcdd54606e4320a5d0f506eb6dabf8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:48.428439 13850 raft_consensus.cc:2258] T dfdcdd54606e4320a5d0f506eb6dabf8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:48.429175 13850 ts_tablet_manager.cc:1875] T dfdcdd54606e4320a5d0f506eb6dabf8 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:48.432111 13850 ts_tablet_manager.cc:1888] T dfdcdd54606e4320a5d0f506eb6dabf8 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 01:59:48.432346 13850 log.cc:1196] T dfdcdd54606e4320a5d0f506eb6dabf8 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/dfdcdd54606e4320a5d0f506eb6dabf8
I20240503 01:59:48.433112 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet dfdcdd54606e4320a5d0f506eb6dabf8 (table rebalance_test_table_19 [id=d2d157b43e4243979ed58dfe4b2fe341]) successfully deleted
I20240503 01:59:48.458204 12366 consensus_queue.cc:1035] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:48.462607 12366 consensus_queue.cc:1035] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:48.485273 6735 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:48.488124 13457 consensus_queue.cc:1035] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:48.501967 13822 tablet.cc:1722] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:48.502331 13822 tablet.cc:1620] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 01:59:48.503288 13826 tablet.cc:1722] T 75643ebd6e5f44e5bc3cbb452514f2ec P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:48.503664 13826 tablet.cc:1620] T 75643ebd6e5f44e5bc3cbb452514f2ec P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:48.507925 13457 consensus_queue.cc:1035] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:48.512917 13823 tablet.cc:1722] T 75643ebd6e5f44e5bc3cbb452514f2ec P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:48.513309 13823 tablet.cc:1620] T 75643ebd6e5f44e5bc3cbb452514f2ec P 5dcc33cc621a4cbbae155f3230fb7c46: MemRowSet was empty: no flush needed.
I20240503 01:59:48.516775 6739 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=865eb68bf451427fa36c8409a20305a8] alter complete (version 2)
I20240503 01:59:48.576408 7256 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:48.581902 9472 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:48.587268 9342 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:48.602442 6993 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:48.603791 6861 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:48.605278 9737 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:48.619235 7124 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:48.619395 9605 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:48.622030 6738 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:48.632818 13822 tablet.cc:1722] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:48.635205 13826 tablet.cc:1722] T 75643ebd6e5f44e5bc3cbb452514f2ec P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:48.646317 13823 tablet.cc:1722] T 75643ebd6e5f44e5bc3cbb452514f2ec P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:48.649873 13434 consensus_queue.cc:1035] T 65b41c24a2504ad386e437c24f1decee P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:48.655938 6733 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=865eb68bf451427fa36c8409a20305a8] alter complete (version 3)
I20240503 01:59:48.659624 6738 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:48.659821 13434 consensus_queue.cc:1035] T 65b41c24a2504ad386e437c24f1decee P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:48.676054 9342 tablet_service.cc:1467] Processing CreateTablet for tablet ca35a5b27ddd40108d7c31e711ca37da (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=70efe522f8fa4d9bb92406e8bfcb0ef3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:48.676048 9472 tablet_service.cc:1467] Processing CreateTablet for tablet ca35a5b27ddd40108d7c31e711ca37da (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=70efe522f8fa4d9bb92406e8bfcb0ef3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:48.676385 9472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca35a5b27ddd40108d7c31e711ca37da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.676394 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca35a5b27ddd40108d7c31e711ca37da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.679410 9737 tablet_service.cc:1467] Processing CreateTablet for tablet ca35a5b27ddd40108d7c31e711ca37da (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=70efe522f8fa4d9bb92406e8bfcb0ef3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:48.679790 9737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca35a5b27ddd40108d7c31e711ca37da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.680657 6993 tablet_service.cc:1467] Processing CreateTablet for tablet 822cbc676b234eda8770f7b485ebe302 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=70efe522f8fa4d9bb92406e8bfcb0ef3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:48.680938 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 822cbc676b234eda8770f7b485ebe302. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.681730 7124 tablet_service.cc:1467] Processing CreateTablet for tablet b886572c6ec9473bbec744cee8de12f5 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=70efe522f8fa4d9bb92406e8bfcb0ef3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:48.682053 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b886572c6ec9473bbec744cee8de12f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.683598 9341 tablet_service.cc:1467] Processing CreateTablet for tablet b886572c6ec9473bbec744cee8de12f5 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=70efe522f8fa4d9bb92406e8bfcb0ef3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:48.683898 9341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b886572c6ec9473bbec744cee8de12f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.686434 9472 tablet_service.cc:1467] Processing CreateTablet for tablet b886572c6ec9473bbec744cee8de12f5 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=70efe522f8fa4d9bb92406e8bfcb0ef3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:48.686805 9472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b886572c6ec9473bbec744cee8de12f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.687624 13923 tablet_bootstrap.cc:492] T b886572c6ec9473bbec744cee8de12f5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:48.688448 13923 tablet_bootstrap.cc:654] T b886572c6ec9473bbec744cee8de12f5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.688758 13924 tablet_bootstrap.cc:492] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:48.689203 13921 tablet_bootstrap.cc:492] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:48.690261 13924 tablet_bootstrap.cc:654] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.691975 13921 tablet_bootstrap.cc:654] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.692221 13924 tablet_bootstrap.cc:492] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:48.692476 13924 ts_tablet_manager.cc:1367] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:48.693710 13924 raft_consensus.cc:348] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:48.693854 13924 raft_consensus.cc:374] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.693897 13924 raft_consensus.cc:729] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.694043 13924 consensus_queue.cc:260] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:48.694377 13924 ts_tablet_manager.cc:1398] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:48.694628 13924 tablet_bootstrap.cc:492] T ca35a5b27ddd40108d7c31e711ca37da P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:48.695744 13924 tablet_bootstrap.cc:654] T ca35a5b27ddd40108d7c31e711ca37da P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.696607 9605 tablet_service.cc:1467] Processing CreateTablet for tablet 822cbc676b234eda8770f7b485ebe302 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=70efe522f8fa4d9bb92406e8bfcb0ef3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:48.696921 9605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 822cbc676b234eda8770f7b485ebe302. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.696944 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 822cbc676b234eda8770f7b485ebe302 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=70efe522f8fa4d9bb92406e8bfcb0ef3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:48.697180 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 822cbc676b234eda8770f7b485ebe302. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.697263 13923 tablet_bootstrap.cc:492] T b886572c6ec9473bbec744cee8de12f5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:48.697364 13923 ts_tablet_manager.cc:1367] T b886572c6ec9473bbec744cee8de12f5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 01:59:48.697448 13920 tablet_bootstrap.cc:492] T ca35a5b27ddd40108d7c31e711ca37da P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:48.698077 13923 raft_consensus.cc:348] T b886572c6ec9473bbec744cee8de12f5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:48.698191 13923 raft_consensus.cc:374] T b886572c6ec9473bbec744cee8de12f5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.698237 13923 raft_consensus.cc:729] T b886572c6ec9473bbec744cee8de12f5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.698383 13923 consensus_queue.cc:260] T b886572c6ec9473bbec744cee8de12f5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:48.698582 13922 tablet_bootstrap.cc:492] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:48.698712 13923 ts_tablet_manager.cc:1398] T b886572c6ec9473bbec744cee8de12f5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 01:59:48.699644 13922 tablet_bootstrap.cc:654] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.700353 13920 tablet_bootstrap.cc:654] T ca35a5b27ddd40108d7c31e711ca37da P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.703100 13929 tablet_bootstrap.cc:492] T 822cbc676b234eda8770f7b485ebe302 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:48.704392 13920 tablet_bootstrap.cc:492] T ca35a5b27ddd40108d7c31e711ca37da P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:48.704537 13920 ts_tablet_manager.cc:1367] T ca35a5b27ddd40108d7c31e711ca37da P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:59:48.706275 13929 tablet_bootstrap.cc:654] T 822cbc676b234eda8770f7b485ebe302 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.706832 13922 tablet_bootstrap.cc:492] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:48.707923 13920 raft_consensus.cc:348] T ca35a5b27ddd40108d7c31e711ca37da P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:48.708256 13922 ts_tablet_manager.cc:1367] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240503 01:59:48.708686 13920 raft_consensus.cc:374] T ca35a5b27ddd40108d7c31e711ca37da P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.708817 13920 raft_consensus.cc:729] T ca35a5b27ddd40108d7c31e711ca37da P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.709066 13920 consensus_queue.cc:260] T ca35a5b27ddd40108d7c31e711ca37da P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:48.709481 13920 ts_tablet_manager.cc:1398] T ca35a5b27ddd40108d7c31e711ca37da P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:48.710143 13922 raft_consensus.cc:348] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:48.710366 13924 tablet_bootstrap.cc:492] T ca35a5b27ddd40108d7c31e711ca37da P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:48.710462 13924 ts_tablet_manager.cc:1367] T ca35a5b27ddd40108d7c31e711ca37da P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 01:59:48.710585 13920 tablet_bootstrap.cc:492] T b886572c6ec9473bbec744cee8de12f5 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:48.710654 13922 raft_consensus.cc:374] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.710726 13922 raft_consensus.cc:729] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.710867 13922 consensus_queue.cc:260] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:48.711098 13924 raft_consensus.cc:348] T ca35a5b27ddd40108d7c31e711ca37da P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:48.711203 13924 raft_consensus.cc:374] T ca35a5b27ddd40108d7c31e711ca37da P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.711233 13924 raft_consensus.cc:729] T ca35a5b27ddd40108d7c31e711ca37da P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.711354 13924 consensus_queue.cc:260] T ca35a5b27ddd40108d7c31e711ca37da P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:48.711680 13924 ts_tablet_manager.cc:1398] T ca35a5b27ddd40108d7c31e711ca37da P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:48.711853 13924 tablet_bootstrap.cc:492] T 822cbc676b234eda8770f7b485ebe302 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:48.712246 13920 tablet_bootstrap.cc:654] T b886572c6ec9473bbec744cee8de12f5 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.712930 13922 ts_tablet_manager.cc:1398] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 01:59:48.712967 13924 tablet_bootstrap.cc:654] T 822cbc676b234eda8770f7b485ebe302 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.714036 13929 tablet_bootstrap.cc:492] T 822cbc676b234eda8770f7b485ebe302 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:48.714162 13929 ts_tablet_manager.cc:1367] T 822cbc676b234eda8770f7b485ebe302 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20240503 01:59:48.714731 13929 raft_consensus.cc:348] T 822cbc676b234eda8770f7b485ebe302 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:48.714972 13929 raft_consensus.cc:374] T 822cbc676b234eda8770f7b485ebe302 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.715085 13929 raft_consensus.cc:729] T 822cbc676b234eda8770f7b485ebe302 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.715314 13929 consensus_queue.cc:260] T 822cbc676b234eda8770f7b485ebe302 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:48.715716 13929 ts_tablet_manager.cc:1398] T 822cbc676b234eda8770f7b485ebe302 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 01:59:48.718497 13924 tablet_bootstrap.cc:492] T 822cbc676b234eda8770f7b485ebe302 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:48.718753 13924 ts_tablet_manager.cc:1367] T 822cbc676b234eda8770f7b485ebe302 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20240503 01:59:48.720044 13924 raft_consensus.cc:348] T 822cbc676b234eda8770f7b485ebe302 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:48.720191 13924 raft_consensus.cc:374] T 822cbc676b234eda8770f7b485ebe302 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.720221 13924 raft_consensus.cc:729] T 822cbc676b234eda8770f7b485ebe302 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.720351 13924 consensus_queue.cc:260] T 822cbc676b234eda8770f7b485ebe302 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:48.721015 13924 ts_tablet_manager.cc:1398] T 822cbc676b234eda8770f7b485ebe302 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:48.722699 13921 tablet_bootstrap.cc:492] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:48.722745 13920 tablet_bootstrap.cc:492] T b886572c6ec9473bbec744cee8de12f5 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:48.722821 13921 ts_tablet_manager.cc:1367] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.001s
I20240503 01:59:48.722831 13920 ts_tablet_manager.cc:1367] T b886572c6ec9473bbec744cee8de12f5 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240503 01:59:48.723337 13920 raft_consensus.cc:348] T b886572c6ec9473bbec744cee8de12f5 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:48.723338 13921 raft_consensus.cc:348] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:48.723454 13921 raft_consensus.cc:374] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.723454 13920 raft_consensus.cc:374] T b886572c6ec9473bbec744cee8de12f5 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.723489 13921 raft_consensus.cc:729] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.723506 13920 raft_consensus.cc:729] T b886572c6ec9473bbec744cee8de12f5 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.723639 13920 consensus_queue.cc:260] T b886572c6ec9473bbec744cee8de12f5 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:48.723639 13921 consensus_queue.cc:260] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:48.723945 13921 ts_tablet_manager.cc:1398] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:48.725364 13920 ts_tablet_manager.cc:1398] T b886572c6ec9473bbec744cee8de12f5 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:48.736805 6882 consensus_queue.cc:237] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:48.738477 9493 raft_consensus.cc:1261] T 8ad236ea5ab347c69026293bef7c04a8 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:48.738504 7274 raft_consensus.cc:1261] T 8ad236ea5ab347c69026293bef7c04a8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:48.738941 13951 consensus_queue.cc:1035] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:48.739216 13385 consensus_queue.cc:1035] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:48.741852 13385 raft_consensus.cc:2932] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:48.745926 9493 raft_consensus.cc:2932] T 8ad236ea5ab347c69026293bef7c04a8 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:48.745930 7274 raft_consensus.cc:2932] T 8ad236ea5ab347c69026293bef7c04a8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:48.748509 6738 catalog_manager.cc:5484] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:48.750694 13349 raft_consensus.cc:482] T ca35a5b27ddd40108d7c31e711ca37da P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:48.750821 13349 raft_consensus.cc:504] T ca35a5b27ddd40108d7c31e711ca37da P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:48.751281 13349 leader_election.cc:288] T ca35a5b27ddd40108d7c31e711ca37da P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:48.751929 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca35a5b27ddd40108d7c31e711ca37da" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:48.752485 9493 raft_consensus.cc:2454] T ca35a5b27ddd40108d7c31e711ca37da P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:48.753216 9296 leader_election.cc:302] T ca35a5b27ddd40108d7c31e711ca37da P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:48.753435 13349 raft_consensus.cc:2781] T ca35a5b27ddd40108d7c31e711ca37da P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:48.753492 13349 raft_consensus.cc:482] T ca35a5b27ddd40108d7c31e711ca37da P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:48.753540 13349 raft_consensus.cc:3037] T ca35a5b27ddd40108d7c31e711ca37da P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.754810 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca35a5b27ddd40108d7c31e711ca37da" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:48.754895 13349 raft_consensus.cc:504] T ca35a5b27ddd40108d7c31e711ca37da P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:48.755064 9757 raft_consensus.cc:2454] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:48.755306 13349 leader_election.cc:288] T ca35a5b27ddd40108d7c31e711ca37da P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Requested vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:48.756276 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca35a5b27ddd40108d7c31e711ca37da" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:48.756498 9493 raft_consensus.cc:3037] T ca35a5b27ddd40108d7c31e711ca37da P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.756907 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca35a5b27ddd40108d7c31e711ca37da" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:48.757069 9757 raft_consensus.cc:3037] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.758250 9757 raft_consensus.cc:2454] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:48.758766 9493 raft_consensus.cc:2454] T ca35a5b27ddd40108d7c31e711ca37da P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:48.758821 7124 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ad236ea5ab347c69026293bef7c04a8 with delete_type TABLET_DATA_TOMBSTONED (TS c688f3df56ac4b6d9b0c30fa4cc8ca2a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:48.759343 9296 leader_election.cc:302] T ca35a5b27ddd40108d7c31e711ca37da P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:48.759567 13349 raft_consensus.cc:2781] T ca35a5b27ddd40108d7c31e711ca37da P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:48.759661 13349 raft_consensus.cc:686] T ca35a5b27ddd40108d7c31e711ca37da P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Running, Role: LEADER
I20240503 01:59:48.759606 6739 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:48.759769 6739 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:48.759810 13349 consensus_queue.cc:237] T ca35a5b27ddd40108d7c31e711ca37da P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:48.760468 13960 tablet_replica.cc:330] T 8ad236ea5ab347c69026293bef7c04a8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:48.760687 13960 raft_consensus.cc:2229] T 8ad236ea5ab347c69026293bef7c04a8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:48.760922 13960 raft_consensus.cc:2258] T 8ad236ea5ab347c69026293bef7c04a8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:48.763525 6739 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 75643ebd6e5f44e5bc3cbb452514f2ec
I20240503 01:59:48.764917 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet 75643ebd6e5f44e5bc3cbb452514f2ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:48 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:48.765111 9737 tablet_service.cc:1514] Processing DeleteTablet for tablet 75643ebd6e5f44e5bc3cbb452514f2ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:48 UTC) from {username='slave'} at 127.0.0.1:48152
I20240503 01:59:48.765152 13850 tablet_replica.cc:330] T 75643ebd6e5f44e5bc3cbb452514f2ec P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:48.765269 9342 tablet_service.cc:1514] Processing DeleteTablet for tablet 75643ebd6e5f44e5bc3cbb452514f2ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:48 UTC) from {username='slave'} at 127.0.0.1:36040
I20240503 01:59:48.765303 13850 raft_consensus.cc:2229] T 75643ebd6e5f44e5bc3cbb452514f2ec P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:48.765437 13850 raft_consensus.cc:2258] T 75643ebd6e5f44e5bc3cbb452514f2ec P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:48.766328 6732 catalog_manager.cc:5484] T ca35a5b27ddd40108d7c31e711ca37da P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:48.766505 13850 ts_tablet_manager.cc:1875] T 75643ebd6e5f44e5bc3cbb452514f2ec P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:48.766999 13960 ts_tablet_manager.cc:1875] T 8ad236ea5ab347c69026293bef7c04a8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:48.769973 13850 ts_tablet_manager.cc:1888] T 75643ebd6e5f44e5bc3cbb452514f2ec P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:48.770174 13850 log.cc:1196] T 75643ebd6e5f44e5bc3cbb452514f2ec P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/75643ebd6e5f44e5bc3cbb452514f2ec
I20240503 01:59:48.770576 13850 ts_tablet_manager.cc:1909] T 75643ebd6e5f44e5bc3cbb452514f2ec P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:48.771836 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 75643ebd6e5f44e5bc3cbb452514f2ec (table rebalancer_dynamic_table_new_name [id=865eb68bf451427fa36c8409a20305a8]) successfully deleted
I20240503 01:59:48.774195 13960 ts_tablet_manager.cc:1888] T 8ad236ea5ab347c69026293bef7c04a8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 01:59:48.774454 13960 log.cc:1196] T 8ad236ea5ab347c69026293bef7c04a8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/8ad236ea5ab347c69026293bef7c04a8
I20240503 01:59:48.775132 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet 8ad236ea5ab347c69026293bef7c04a8 (table rebalance_test_table_13 [id=06feff591c384a50825820a47b49c8f9]) successfully deleted
I20240503 01:59:48.775220 13962 tablet_replica.cc:330] T 75643ebd6e5f44e5bc3cbb452514f2ec P db1a0000aaa346f79c0aa4b6c5105191: stopping tablet replica
I20240503 01:59:48.775406 13962 raft_consensus.cc:2229] T 75643ebd6e5f44e5bc3cbb452514f2ec P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:48.775545 13962 raft_consensus.cc:2258] T 75643ebd6e5f44e5bc3cbb452514f2ec P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:48.776279 13962 ts_tablet_manager.cc:1875] T 75643ebd6e5f44e5bc3cbb452514f2ec P db1a0000aaa346f79c0aa4b6c5105191: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:48.782289 13962 ts_tablet_manager.cc:1888] T 75643ebd6e5f44e5bc3cbb452514f2ec P db1a0000aaa346f79c0aa4b6c5105191: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:48.782555 13962 log.cc:1196] T 75643ebd6e5f44e5bc3cbb452514f2ec P db1a0000aaa346f79c0aa4b6c5105191: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/75643ebd6e5f44e5bc3cbb452514f2ec
I20240503 01:59:48.782712 13963 tablet_replica.cc:330] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8: stopping tablet replica
I20240503 01:59:48.782949 13963 raft_consensus.cc:2229] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:48.783169 13962 ts_tablet_manager.cc:1909] T 75643ebd6e5f44e5bc3cbb452514f2ec P db1a0000aaa346f79c0aa4b6c5105191: Deleting consensus metadata
I20240503 01:59:48.783270 13963 raft_consensus.cc:2258] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:48.784010 13963 ts_tablet_manager.cc:1875] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:48.784240 6726 catalog_manager.cc:4830] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): tablet 75643ebd6e5f44e5bc3cbb452514f2ec (table rebalancer_dynamic_table_new_name [id=865eb68bf451427fa36c8409a20305a8]) successfully deleted
I20240503 01:59:48.796653 13963 ts_tablet_manager.cc:1888] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:48.796763 13963 log.cc:1196] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/75643ebd6e5f44e5bc3cbb452514f2ec
I20240503 01:59:48.797049 13963 ts_tablet_manager.cc:1909] T 75643ebd6e5f44e5bc3cbb452514f2ec P 12e8c683db054df3b75c6285b98a5ad8: Deleting consensus metadata
I20240503 01:59:48.798060 6727 catalog_manager.cc:4830] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): tablet 75643ebd6e5f44e5bc3cbb452514f2ec (table rebalancer_dynamic_table_new_name [id=865eb68bf451427fa36c8409a20305a8]) successfully deleted
I20240503 01:59:48.865528 6740 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:48.874209 9605 tablet_service.cc:1467] Processing CreateTablet for tablet a3be7c7a52ee4569be546ee48caaa4ff (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c587e31d48724de8bf73191bfe2ae533]), partition=
I20240503 01:59:48.874464 9737 tablet_service.cc:1467] Processing CreateTablet for tablet a3be7c7a52ee4569be546ee48caaa4ff (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c587e31d48724de8bf73191bfe2ae533]), partition=
I20240503 01:59:48.874498 9605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3be7c7a52ee4569be546ee48caaa4ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.874722 9737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3be7c7a52ee4569be546ee48caaa4ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.874910 9472 tablet_service.cc:1467] Processing CreateTablet for tablet a3be7c7a52ee4569be546ee48caaa4ff (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c587e31d48724de8bf73191bfe2ae533]), partition=
I20240503 01:59:48.875142 9472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3be7c7a52ee4569be546ee48caaa4ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:48.876879 13921 tablet_bootstrap.cc:492] T a3be7c7a52ee4569be546ee48caaa4ff P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:48.876966 13929 tablet_bootstrap.cc:492] T a3be7c7a52ee4569be546ee48caaa4ff P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:48.877133 13920 tablet_bootstrap.cc:492] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:48.877998 13921 tablet_bootstrap.cc:654] T a3be7c7a52ee4569be546ee48caaa4ff P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.877993 13920 tablet_bootstrap.cc:654] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.877990 13929 tablet_bootstrap.cc:654] T a3be7c7a52ee4569be546ee48caaa4ff P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:48.879583 13929 tablet_bootstrap.cc:492] T a3be7c7a52ee4569be546ee48caaa4ff P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:48.880332 13920 tablet_bootstrap.cc:492] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:48.880430 13920 ts_tablet_manager.cc:1367] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:59:48.880421 13929 ts_tablet_manager.cc:1367] T a3be7c7a52ee4569be546ee48caaa4ff P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:59:48.880913 13921 tablet_bootstrap.cc:492] T a3be7c7a52ee4569be546ee48caaa4ff P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:48.881006 13921 ts_tablet_manager.cc:1367] T a3be7c7a52ee4569be546ee48caaa4ff P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 01:59:48.880966 13920 raft_consensus.cc:348] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:48.881083 13920 raft_consensus.cc:374] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.881124 13920 raft_consensus.cc:729] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.881155 13929 raft_consensus.cc:348] T a3be7c7a52ee4569be546ee48caaa4ff P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:48.881278 13929 raft_consensus.cc:374] T a3be7c7a52ee4569be546ee48caaa4ff P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.881253 13920 consensus_queue.cc:260] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:48.881316 13929 raft_consensus.cc:729] T a3be7c7a52ee4569be546ee48caaa4ff P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.881441 13929 consensus_queue.cc:260] T a3be7c7a52ee4569be546ee48caaa4ff P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:48.881579 13920 ts_tablet_manager.cc:1398] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:48.881520 13921 raft_consensus.cc:348] T a3be7c7a52ee4569be546ee48caaa4ff P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:48.881645 13921 raft_consensus.cc:374] T a3be7c7a52ee4569be546ee48caaa4ff P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:48.881678 13921 raft_consensus.cc:729] T a3be7c7a52ee4569be546ee48caaa4ff P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:48.881712 13929 ts_tablet_manager.cc:1398] T a3be7c7a52ee4569be546ee48caaa4ff P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:48.881809 13921 consensus_queue.cc:260] T a3be7c7a52ee4569be546ee48caaa4ff P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:48.882090 13921 ts_tablet_manager.cc:1398] T a3be7c7a52ee4569be546ee48caaa4ff P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:48.983568 13457 raft_consensus.cc:482] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:48.983810 13457 raft_consensus.cc:504] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:48.984501 13457 leader_election.cc:288] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:48.985031 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b886572c6ec9473bbec744cee8de12f5" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:48.985095 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b886572c6ec9473bbec744cee8de12f5" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:48.985227 9493 raft_consensus.cc:2454] T b886572c6ec9473bbec744cee8de12f5 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:48.985226 7143 raft_consensus.cc:2454] T b886572c6ec9473bbec744cee8de12f5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:48.985690 9296 leader_election.cc:302] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:48.985997 13457 raft_consensus.cc:2781] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:48.986052 13457 raft_consensus.cc:482] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:48.986099 13457 raft_consensus.cc:3037] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.987319 13457 raft_consensus.cc:504] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:48.987787 13457 leader_election.cc:288] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Requested vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:48.988373 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b886572c6ec9473bbec744cee8de12f5" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:48.988415 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b886572c6ec9473bbec744cee8de12f5" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:48.988509 9493 raft_consensus.cc:3037] T b886572c6ec9473bbec744cee8de12f5 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.988569 7143 raft_consensus.cc:3037] T b886572c6ec9473bbec744cee8de12f5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:48.989779 7143 raft_consensus.cc:2454] T b886572c6ec9473bbec744cee8de12f5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:48.989939 9493 raft_consensus.cc:2454] T b886572c6ec9473bbec744cee8de12f5 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:48.990111 9298 leader_election.cc:302] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:48.990315 13457 raft_consensus.cc:2781] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:48.990423 13457 raft_consensus.cc:686] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Running, Role: LEADER
I20240503 01:59:48.990639 13457 consensus_queue.cc:237] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:48.993105 6733 catalog_manager.cc:5484] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:49.007094 13636 raft_consensus.cc:482] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:49.007280 13636 raft_consensus.cc:504] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:49.008008 13636 leader_election.cc:288] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:49.008569 9362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "822cbc676b234eda8770f7b485ebe302" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:49.008698 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "822cbc676b234eda8770f7b485ebe302" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:49.008808 9362 raft_consensus.cc:2454] T 822cbc676b234eda8770f7b485ebe302 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:49.008872 9625 raft_consensus.cc:2454] T 822cbc676b234eda8770f7b485ebe302 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:49.009203 6949 leader_election.cc:302] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:49.009485 13636 raft_consensus.cc:2781] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:49.009562 13636 raft_consensus.cc:482] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:49.009599 13636 raft_consensus.cc:3037] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.010859 13636 raft_consensus.cc:504] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:49.011355 13636 leader_election.cc:288] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:49.012092 9362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "822cbc676b234eda8770f7b485ebe302" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:49.012274 9362 raft_consensus.cc:3037] T 822cbc676b234eda8770f7b485ebe302 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.012377 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "822cbc676b234eda8770f7b485ebe302" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:49.012667 9625 raft_consensus.cc:3037] T 822cbc676b234eda8770f7b485ebe302 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.013639 9625 raft_consensus.cc:2454] T 822cbc676b234eda8770f7b485ebe302 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:49.013708 9362 raft_consensus.cc:2454] T 822cbc676b234eda8770f7b485ebe302 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:49.014113 6949 leader_election.cc:302] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:49.014421 12321 raft_consensus.cc:2781] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:49.014765 12321 raft_consensus.cc:686] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:49.015008 12321 consensus_queue.cc:237] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:49.018247 6733 catalog_manager.cc:5484] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:49.032182 13671 raft_consensus.cc:482] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:49.032332 13671 raft_consensus.cc:504] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:49.032986 13671 leader_election.cc:288] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:49.033411 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3be7c7a52ee4569be546ee48caaa4ff" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:49.033566 9625 raft_consensus.cc:2454] T a3be7c7a52ee4569be546ee48caaa4ff P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:49.033771 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3be7c7a52ee4569be546ee48caaa4ff" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:49.033830 9429 leader_election.cc:302] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:49.033923 9757 raft_consensus.cc:2454] T a3be7c7a52ee4569be546ee48caaa4ff P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:49.034018 13671 raft_consensus.cc:2781] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:49.034077 13671 raft_consensus.cc:482] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:49.034122 13671 raft_consensus.cc:3037] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.035437 13671 raft_consensus.cc:504] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:49.035917 13671 leader_election.cc:288] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:49.036425 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3be7c7a52ee4569be546ee48caaa4ff" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:49.036406 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3be7c7a52ee4569be546ee48caaa4ff" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:49.036553 9757 raft_consensus.cc:3037] T a3be7c7a52ee4569be546ee48caaa4ff P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.037822 9757 raft_consensus.cc:2454] T a3be7c7a52ee4569be546ee48caaa4ff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:49.038156 9625 raft_consensus.cc:3037] T a3be7c7a52ee4569be546ee48caaa4ff P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.038215 9428 leader_election.cc:302] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:49.038398 13671 raft_consensus.cc:2781] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:49.039410 9625 raft_consensus.cc:2454] T a3be7c7a52ee4569be546ee48caaa4ff P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:49.039906 13671 raft_consensus.cc:686] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Becoming Leader. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Running, Role: LEADER
I20240503 01:59:49.040184 13671 consensus_queue.cc:237] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:49.044361 6733 catalog_manager.cc:5484] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: term changed from 0 to 1, leader changed from <none> to bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:49.057050 9342 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:49.067611 7256 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:49.067771 6861 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:49.071175 9605 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:49.078173 9472 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:49.083843 7124 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:49.098724 9737 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:49.101712 6993 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:49.173954 13457 consensus_queue.cc:1035] T ca35a5b27ddd40108d7c31e711ca37da P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:49.178882 6740 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:49.184434 13349 consensus_queue.cc:1035] T ca35a5b27ddd40108d7c31e711ca37da P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:49.191633 9757 raft_consensus.cc:1261] T a3be7c7a52ee4569be546ee48caaa4ff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:49.192689 13671 consensus_queue.cc:1035] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:49.194332 9625 raft_consensus.cc:1261] T a3be7c7a52ee4569be546ee48caaa4ff P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:49.194906 13671 consensus_queue.cc:1035] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:49.196329 6733 catalog_manager.cc:5576] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053) does not have the latest schema for tablet a3be7c7a52ee4569be546ee48caaa4ff (table rebalancer_dynamic_table [id=c587e31d48724de8bf73191bfe2ae533]). Expected version 1 got 0
I20240503 01:59:49.198254 6733 catalog_manager.cc:5576] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087) does not have the latest schema for tablet a3be7c7a52ee4569be546ee48caaa4ff (table rebalancer_dynamic_table [id=c587e31d48724de8bf73191bfe2ae533]). Expected version 1 got 0
I20240503 01:59:49.199530 6733 catalog_manager.cc:5576] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053) does not have the latest schema for tablet a3be7c7a52ee4569be546ee48caaa4ff (table rebalancer_dynamic_table [id=c587e31d48724de8bf73191bfe2ae533]). Expected version 1 got 0
I20240503 01:59:49.200444 14035 tablet.cc:1722] T a3be7c7a52ee4569be546ee48caaa4ff P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:49.200878 14035 tablet.cc:1620] T a3be7c7a52ee4569be546ee48caaa4ff P b7062fbbc2b54fff9e99ec4ea5562c76: MemRowSet was empty: no flush needed.
I20240503 01:59:49.201929 6733 catalog_manager.cc:5576] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087) does not have the latest schema for tablet a3be7c7a52ee4569be546ee48caaa4ff (table rebalancer_dynamic_table [id=c587e31d48724de8bf73191bfe2ae533]). Expected version 1 got 0
I20240503 01:59:49.202003 14033 tablet.cc:1722] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:49.202472 14033 tablet.cc:1620] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:49.204517 14034 tablet.cc:1722] T a3be7c7a52ee4569be546ee48caaa4ff P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:49.204874 14034 tablet.cc:1620] T a3be7c7a52ee4569be546ee48caaa4ff P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:49.208791 6733 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=c587e31d48724de8bf73191bfe2ae533] alter complete (version 1)
I20240503 01:59:49.224952 9493 consensus_queue.cc:237] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:49.226343 7014 raft_consensus.cc:1261] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:49.226507 9625 raft_consensus.cc:1261] T 31f5289f1dff4fcaa3492c16e40a3ed7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:49.226861 13671 consensus_queue.cc:1035] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:49.227090 14006 consensus_queue.cc:1035] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:49.229013 14049 raft_consensus.cc:2932] T 31f5289f1dff4fcaa3492c16e40a3ed7 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:49.229456 7014 raft_consensus.cc:2932] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:49.229509 9625 raft_consensus.cc:2932] T 31f5289f1dff4fcaa3492c16e40a3ed7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:49.233274 6737 catalog_manager.cc:5484] T 31f5289f1dff4fcaa3492c16e40a3ed7 P b7062fbbc2b54fff9e99ec4ea5562c76 reported cstate change: config changed from index 2 to 3, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:49.237247 9737 tablet_service.cc:1514] Processing DeleteTablet for tablet 31f5289f1dff4fcaa3492c16e40a3ed7 with delete_type TABLET_DATA_TOMBSTONED (TS db1a0000aaa346f79c0aa4b6c5105191 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:48152
I20240503 01:59:49.237576 13962 tablet_replica.cc:330] T 31f5289f1dff4fcaa3492c16e40a3ed7 P db1a0000aaa346f79c0aa4b6c5105191: stopping tablet replica
I20240503 01:59:49.237748 13962 raft_consensus.cc:2229] T 31f5289f1dff4fcaa3492c16e40a3ed7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:49.237908 13962 raft_consensus.cc:2258] T 31f5289f1dff4fcaa3492c16e40a3ed7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:49.238391 13962 ts_tablet_manager.cc:1875] T 31f5289f1dff4fcaa3492c16e40a3ed7 P db1a0000aaa346f79c0aa4b6c5105191: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:49.240860 13962 ts_tablet_manager.cc:1888] T 31f5289f1dff4fcaa3492c16e40a3ed7 P db1a0000aaa346f79c0aa4b6c5105191: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 01:59:49.241075 13962 log.cc:1196] T 31f5289f1dff4fcaa3492c16e40a3ed7 P db1a0000aaa346f79c0aa4b6c5105191: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/31f5289f1dff4fcaa3492c16e40a3ed7
I20240503 01:59:49.242298 6726 catalog_manager.cc:4830] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): tablet 31f5289f1dff4fcaa3492c16e40a3ed7 (table rebalancer_extra_table_8 [id=5094b51906a7434daee33202abd81bfc]) successfully deleted
I20240503 01:59:49.314306 6737 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:49.323276 14033 tablet.cc:1722] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:49.323674 14033 tablet.cc:1620] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:49.323777 14035 tablet.cc:1722] T a3be7c7a52ee4569be546ee48caaa4ff P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:49.324105 14035 tablet.cc:1620] T a3be7c7a52ee4569be546ee48caaa4ff P b7062fbbc2b54fff9e99ec4ea5562c76: MemRowSet was empty: no flush needed.
I20240503 01:59:49.324095 14034 tablet.cc:1722] T a3be7c7a52ee4569be546ee48caaa4ff P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:49.324434 14034 tablet.cc:1620] T a3be7c7a52ee4569be546ee48caaa4ff P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:49.332542 6735 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=c587e31d48724de8bf73191bfe2ae533] alter complete (version 2)
I20240503 01:59:49.378429 13457 consensus_queue.cc:1035] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:49.385591 13349 consensus_queue.cc:1035] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:49.422618 7256 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:49.423763 7124 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:49.426950 6861 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:49.427994 13636 consensus_queue.cc:1035] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:49.431058 9342 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:49.432407 6993 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:49.440302 6740 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:49.441147 9472 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 01:59:49.442405 14004 consensus_queue.cc:1035] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:49.449441 9605 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:49.455711 9737 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:49.466609 14033 tablet.cc:1722] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:49.470219 14034 tablet.cc:1722] T a3be7c7a52ee4569be546ee48caaa4ff P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:49.473260 14035 tablet.cc:1722] T a3be7c7a52ee4569be546ee48caaa4ff P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:49.473470 6740 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=c587e31d48724de8bf73191bfe2ae533] alter complete (version 3)
I20240503 01:59:49.565892 6733 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:49.575026 9625 consensus_queue.cc:237] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:49.575754 7124 tablet_service.cc:1467] Processing CreateTablet for tablet 8c97333c56ef4c72ac656d4a0a487c80 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6ff0348b1ff345cabc0f4ddf7dd75e21]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:49.576128 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c97333c56ef4c72ac656d4a0a487c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.576391 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 02e8cbe02d1b4017909e9bfddaf093d8 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6ff0348b1ff345cabc0f4ddf7dd75e21]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:49.576705 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02e8cbe02d1b4017909e9bfddaf093d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.577009 9735 tablet_service.cc:1467] Processing CreateTablet for tablet 02e8cbe02d1b4017909e9bfddaf093d8 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6ff0348b1ff345cabc0f4ddf7dd75e21]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:49.577314 9735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02e8cbe02d1b4017909e9bfddaf093d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.578920 9736 tablet_service.cc:1467] Processing CreateTablet for tablet f2d66da6d2e24531aee4aaa6740a7af4 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6ff0348b1ff345cabc0f4ddf7dd75e21]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:49.579208 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2d66da6d2e24531aee4aaa6740a7af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.579617 9737 tablet_service.cc:1467] Processing CreateTablet for tablet 8c97333c56ef4c72ac656d4a0a487c80 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6ff0348b1ff345cabc0f4ddf7dd75e21]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:49.579627 7256 tablet_service.cc:1467] Processing CreateTablet for tablet 02e8cbe02d1b4017909e9bfddaf093d8 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6ff0348b1ff345cabc0f4ddf7dd75e21]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:49.579737 14143 tablet_bootstrap.cc:492] T 02e8cbe02d1b4017909e9bfddaf093d8 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:49.579912 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02e8cbe02d1b4017909e9bfddaf093d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.580605 6861 tablet_service.cc:1467] Processing CreateTablet for tablet f2d66da6d2e24531aee4aaa6740a7af4 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6ff0348b1ff345cabc0f4ddf7dd75e21]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:49.580785 14143 tablet_bootstrap.cc:654] T 02e8cbe02d1b4017909e9bfddaf093d8 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:49.580870 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2d66da6d2e24531aee4aaa6740a7af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.582938 14145 tablet_bootstrap.cc:492] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:49.583086 7275 raft_consensus.cc:1261] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer b7062fbbc2b54fff9e99ec4ea5562c76: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:49.583492 9755 raft_consensus.cc:1261] T 1d182ce2c5084082a0990ce8090eb88b P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer b7062fbbc2b54fff9e99ec4ea5562c76: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:49.583674 14143 tablet_bootstrap.cc:492] T 02e8cbe02d1b4017909e9bfddaf093d8 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:49.583629 14142 consensus_queue.cc:1035] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:49.583760 14143 ts_tablet_manager.cc:1367] T 02e8cbe02d1b4017909e9bfddaf093d8 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:49.585011 9473 tablet_service.cc:1467] Processing CreateTablet for tablet f2d66da6d2e24531aee4aaa6740a7af4 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6ff0348b1ff345cabc0f4ddf7dd75e21]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:49.585011 9472 tablet_service.cc:1467] Processing CreateTablet for tablet 8c97333c56ef4c72ac656d4a0a487c80 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6ff0348b1ff345cabc0f4ddf7dd75e21]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:49.585418 9472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c97333c56ef4c72ac656d4a0a487c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.585372 14143 raft_consensus.cc:348] T 02e8cbe02d1b4017909e9bfddaf093d8 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:49.585501 14143 raft_consensus.cc:374] T 02e8cbe02d1b4017909e9bfddaf093d8 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:49.585541 14143 raft_consensus.cc:729] T 02e8cbe02d1b4017909e9bfddaf093d8 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:49.585665 14143 consensus_queue.cc:260] T 02e8cbe02d1b4017909e9bfddaf093d8 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:49.585981 14143 ts_tablet_manager.cc:1398] T 02e8cbe02d1b4017909e9bfddaf093d8 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:49.586977 14143 tablet_bootstrap.cc:492] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:49.588089 14143 tablet_bootstrap.cc:654] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:49.588171 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2d66da6d2e24531aee4aaa6740a7af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.588866 14148 tablet_bootstrap.cc:492] T 8c97333c56ef4c72ac656d4a0a487c80 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:49.589715 14145 tablet_bootstrap.cc:654] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:49.591658 9737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c97333c56ef4c72ac656d4a0a487c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.591830 14143 tablet_bootstrap.cc:492] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:49.591922 14143 ts_tablet_manager.cc:1367] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:49.592092 14148 tablet_bootstrap.cc:654] T 8c97333c56ef4c72ac656d4a0a487c80 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:49.592392 14143 raft_consensus.cc:348] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:49.592516 14143 raft_consensus.cc:374] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:49.592547 14143 raft_consensus.cc:729] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:49.592679 14143 consensus_queue.cc:260] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:49.592994 14143 ts_tablet_manager.cc:1398] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:49.594061 14145 tablet_bootstrap.cc:492] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:49.594137 14148 tablet_bootstrap.cc:492] T 8c97333c56ef4c72ac656d4a0a487c80 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:49.594172 14145 ts_tablet_manager.cc:1367] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 01:59:49.594271 14148 ts_tablet_manager.cc:1367] T 8c97333c56ef4c72ac656d4a0a487c80 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:49.595434 13434 consensus_queue.cc:1035] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:49.595789 14145 raft_consensus.cc:348] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:49.595928 14145 raft_consensus.cc:374] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:49.595961 14145 raft_consensus.cc:729] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:49.596081 14145 consensus_queue.cc:260] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:49.596997 14148 raft_consensus.cc:348] T 8c97333c56ef4c72ac656d4a0a487c80 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:49.597328 14145 ts_tablet_manager.cc:1398] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:49.598536 14148 raft_consensus.cc:374] T 8c97333c56ef4c72ac656d4a0a487c80 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:49.598644 14148 raft_consensus.cc:729] T 8c97333c56ef4c72ac656d4a0a487c80 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:49.599212 14148 consensus_queue.cc:260] T 8c97333c56ef4c72ac656d4a0a487c80 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:49.599673 6733 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:49.599835 6733 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:49.599972 14148 ts_tablet_manager.cc:1398] T 8c97333c56ef4c72ac656d4a0a487c80 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240503 01:59:49.600279 14148 tablet_bootstrap.cc:492] T f2d66da6d2e24531aee4aaa6740a7af4 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:49.601320 14142 raft_consensus.cc:2932] T 1d182ce2c5084082a0990ce8090eb88b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:49.601909 14148 tablet_bootstrap.cc:654] T f2d66da6d2e24531aee4aaa6740a7af4 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:49.603402 9755 raft_consensus.cc:2932] T 1d182ce2c5084082a0990ce8090eb88b P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:49.603929 14148 tablet_bootstrap.cc:492] T f2d66da6d2e24531aee4aaa6740a7af4 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:49.603931 6733 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet a3be7c7a52ee4569be546ee48caaa4ff
I20240503 01:59:49.604032 14148 ts_tablet_manager.cc:1367] T f2d66da6d2e24531aee4aaa6740a7af4 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 01:59:49.604872 14148 raft_consensus.cc:348] T f2d66da6d2e24531aee4aaa6740a7af4 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:49.605047 14148 raft_consensus.cc:374] T f2d66da6d2e24531aee4aaa6740a7af4 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:49.605090 14148 raft_consensus.cc:729] T f2d66da6d2e24531aee4aaa6740a7af4 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:49.605216 14148 consensus_queue.cc:260] T f2d66da6d2e24531aee4aaa6740a7af4 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:49.605582 14148 ts_tablet_manager.cc:1398] T f2d66da6d2e24531aee4aaa6740a7af4 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 01:59:49.605840 9736 tablet_service.cc:1514] Processing DeleteTablet for tablet a3be7c7a52ee4569be546ee48caaa4ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:49 UTC) from {username='slave'} at 127.0.0.1:48152
I20240503 01:59:49.606130 13962 tablet_replica.cc:330] T a3be7c7a52ee4569be546ee48caaa4ff P db1a0000aaa346f79c0aa4b6c5105191: stopping tablet replica
I20240503 01:59:49.606266 13962 raft_consensus.cc:2229] T a3be7c7a52ee4569be546ee48caaa4ff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:49.606392 13962 raft_consensus.cc:2258] T a3be7c7a52ee4569be546ee48caaa4ff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:49.606724 6733 catalog_manager.cc:5484] T 1d182ce2c5084082a0990ce8090eb88b P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:49.607050 9605 tablet_service.cc:1514] Processing DeleteTablet for tablet a3be7c7a52ee4569be546ee48caaa4ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:49 UTC) from {username='slave'} at 127.0.0.1:58562
I20240503 01:59:49.607385 13962 ts_tablet_manager.cc:1875] T a3be7c7a52ee4569be546ee48caaa4ff P db1a0000aaa346f79c0aa4b6c5105191: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:49.607775 14157 tablet_bootstrap.cc:492] T f2d66da6d2e24531aee4aaa6740a7af4 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:49.609598 14157 tablet_bootstrap.cc:654] T f2d66da6d2e24531aee4aaa6740a7af4 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:49.610937 9342 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d182ce2c5084082a0990ce8090eb88b with delete_type TABLET_DATA_TOMBSTONED (TS 12e8c683db054df3b75c6285b98a5ad8 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:36040
I20240503 01:59:49.612545 13962 ts_tablet_manager.cc:1888] T a3be7c7a52ee4569be546ee48caaa4ff P db1a0000aaa346f79c0aa4b6c5105191: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:49.612672 13962 log.cc:1196] T a3be7c7a52ee4569be546ee48caaa4ff P db1a0000aaa346f79c0aa4b6c5105191: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/a3be7c7a52ee4569be546ee48caaa4ff
I20240503 01:59:49.612737 9473 tablet_service.cc:1514] Processing DeleteTablet for tablet a3be7c7a52ee4569be546ee48caaa4ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:49 UTC) from {username='slave'} at 127.0.0.1:54678
I20240503 01:59:49.613472 14160 tablet_replica.cc:330] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a: stopping tablet replica
I20240503 01:59:49.614334 14160 raft_consensus.cc:2229] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:49.614616 14143 tablet_bootstrap.cc:492] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:49.614900 14160 raft_consensus.cc:2258] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:49.615942 14143 tablet_bootstrap.cc:654] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:49.616288 14146 tablet_bootstrap.cc:492] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:49.616945 7275 raft_consensus.cc:2932] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:49.620040 14160 ts_tablet_manager.cc:1875] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:49.621651 14146 tablet_bootstrap.cc:654] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:49.622068 13962 ts_tablet_manager.cc:1909] T a3be7c7a52ee4569be546ee48caaa4ff P db1a0000aaa346f79c0aa4b6c5105191: Deleting consensus metadata
I20240503 01:59:49.622923 14160 ts_tablet_manager.cc:1888] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:49.622987 14146 tablet_bootstrap.cc:492] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:49.623051 6726 catalog_manager.cc:4830] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): tablet a3be7c7a52ee4569be546ee48caaa4ff (table rebalancer_dynamic_table_new_name [id=c587e31d48724de8bf73191bfe2ae533]) successfully deleted
I20240503 01:59:49.623099 14146 ts_tablet_manager.cc:1367] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:59:49.623215 14160 log.cc:1196] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/a3be7c7a52ee4569be546ee48caaa4ff
I20240503 01:59:49.623478 14160 ts_tablet_manager.cc:1909] T a3be7c7a52ee4569be546ee48caaa4ff P bb5364d730cb4489a0bfc6c270b7450a: Deleting consensus metadata
I20240503 01:59:49.623605 14156 tablet_replica.cc:330] T a3be7c7a52ee4569be546ee48caaa4ff P b7062fbbc2b54fff9e99ec4ea5562c76: stopping tablet replica
I20240503 01:59:49.623822 14156 raft_consensus.cc:2229] T a3be7c7a52ee4569be546ee48caaa4ff P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:49.623960 14146 raft_consensus.cc:348] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:49.624095 14146 raft_consensus.cc:374] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:49.624140 14146 raft_consensus.cc:729] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:49.624280 14156 raft_consensus.cc:2258] T a3be7c7a52ee4569be546ee48caaa4ff P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:49.624289 14146 consensus_queue.cc:260] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:49.624589 14146 ts_tablet_manager.cc:1398] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:49.624876 6725 catalog_manager.cc:4830] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): tablet a3be7c7a52ee4569be546ee48caaa4ff (table rebalancer_dynamic_table_new_name [id=c587e31d48724de8bf73191bfe2ae533]) successfully deleted
I20240503 01:59:49.625445 14143 tablet_bootstrap.cc:492] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:49.625448 14161 tablet_bootstrap.cc:492] T 8c97333c56ef4c72ac656d4a0a487c80 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:49.625550 14143 ts_tablet_manager.cc:1367] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240503 01:59:49.625959 14156 ts_tablet_manager.cc:1875] T a3be7c7a52ee4569be546ee48caaa4ff P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:49.626091 14143 raft_consensus.cc:348] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:49.626194 14143 raft_consensus.cc:374] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:49.626235 14143 raft_consensus.cc:729] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:49.626302 14161 tablet_bootstrap.cc:654] T 8c97333c56ef4c72ac656d4a0a487c80 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:49.626389 14143 consensus_queue.cc:260] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:49.627610 14161 tablet_bootstrap.cc:492] T 8c97333c56ef4c72ac656d4a0a487c80 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:49.627727 14161 ts_tablet_manager.cc:1367] T 8c97333c56ef4c72ac656d4a0a487c80 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:49.628301 14161 raft_consensus.cc:348] T 8c97333c56ef4c72ac656d4a0a487c80 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:49.628424 14161 raft_consensus.cc:374] T 8c97333c56ef4c72ac656d4a0a487c80 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:49.628451 14161 raft_consensus.cc:729] T 8c97333c56ef4c72ac656d4a0a487c80 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:49.628613 14161 consensus_queue.cc:260] T 8c97333c56ef4c72ac656d4a0a487c80 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:49.628839 14143 ts_tablet_manager.cc:1398] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:49.628883 14159 tablet_replica.cc:330] T 1d182ce2c5084082a0990ce8090eb88b P 12e8c683db054df3b75c6285b98a5ad8: stopping tablet replica
I20240503 01:59:49.628949 14161 ts_tablet_manager.cc:1398] T 8c97333c56ef4c72ac656d4a0a487c80 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:49.629043 14159 raft_consensus.cc:2229] T 1d182ce2c5084082a0990ce8090eb88b P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:49.629174 14159 raft_consensus.cc:2258] T 1d182ce2c5084082a0990ce8090eb88b P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:49.631279 14159 ts_tablet_manager.cc:1875] T 1d182ce2c5084082a0990ce8090eb88b P 12e8c683db054df3b75c6285b98a5ad8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:49.635607 13349 raft_consensus.cc:482] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:49.635746 13349 raft_consensus.cc:504] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:49.636237 13349 leader_election.cc:288] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:49.636461 14157 tablet_bootstrap.cc:492] T f2d66da6d2e24531aee4aaa6740a7af4 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:49.636610 14157 ts_tablet_manager.cc:1367] T f2d66da6d2e24531aee4aaa6740a7af4 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20240503 01:59:49.638197 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02e8cbe02d1b4017909e9bfddaf093d8" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:49.638202 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02e8cbe02d1b4017909e9bfddaf093d8" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:49.638394 7275 raft_consensus.cc:2454] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:49.638394 9755 raft_consensus.cc:2454] T 02e8cbe02d1b4017909e9bfddaf093d8 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:49.637764 14157 raft_consensus.cc:348] T f2d66da6d2e24531aee4aaa6740a7af4 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:49.638907 14157 raft_consensus.cc:374] T f2d66da6d2e24531aee4aaa6740a7af4 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:49.639117 9297 leader_election.cc:302] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 1d0e738c34af40fa9950c6f086619b3a; no voters:
I20240503 01:59:49.639192 14157 raft_consensus.cc:729] T f2d66da6d2e24531aee4aaa6740a7af4 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:49.639219 14159 ts_tablet_manager.cc:1888] T 1d182ce2c5084082a0990ce8090eb88b P 12e8c683db054df3b75c6285b98a5ad8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 01:59:49.639353 14159 log.cc:1196] T 1d182ce2c5084082a0990ce8090eb88b P 12e8c683db054df3b75c6285b98a5ad8: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/1d182ce2c5084082a0990ce8090eb88b
I20240503 01:59:49.639392 14157 consensus_queue.cc:260] T f2d66da6d2e24531aee4aaa6740a7af4 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:49.639532 13457 raft_consensus.cc:2781] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:49.639608 13457 raft_consensus.cc:482] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:49.639658 13457 raft_consensus.cc:3037] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.639958 6727 catalog_manager.cc:4830] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): tablet 1d182ce2c5084082a0990ce8090eb88b (table rebalancer_extra_table_2 [id=cbc39ac1f52f44debbf6d5a5d0edd3ef]) successfully deleted
I20240503 01:59:49.640699 14157 ts_tablet_manager.cc:1398] T f2d66da6d2e24531aee4aaa6740a7af4 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:49.640828 13457 raft_consensus.cc:504] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:49.641227 13457 leader_election.cc:288] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Requested vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:49.642827 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02e8cbe02d1b4017909e9bfddaf093d8" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:49.642946 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02e8cbe02d1b4017909e9bfddaf093d8" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:49.643023 7275 raft_consensus.cc:3037] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.643095 9755 raft_consensus.cc:3037] T 02e8cbe02d1b4017909e9bfddaf093d8 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.644382 9755 raft_consensus.cc:2454] T 02e8cbe02d1b4017909e9bfddaf093d8 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:49.644440 7275 raft_consensus.cc:2454] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:49.644555 14156 ts_tablet_manager.cc:1888] T a3be7c7a52ee4569be546ee48caaa4ff P b7062fbbc2b54fff9e99ec4ea5562c76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:49.644666 14156 log.cc:1196] T a3be7c7a52ee4569be546ee48caaa4ff P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/a3be7c7a52ee4569be546ee48caaa4ff
I20240503 01:59:49.645089 9297 leader_election.cc:302] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 1d0e738c34af40fa9950c6f086619b3a; no voters:
I20240503 01:59:49.645658 13349 raft_consensus.cc:2781] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:49.645756 13349 raft_consensus.cc:686] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Running, Role: LEADER
I20240503 01:59:49.645888 14156 ts_tablet_manager.cc:1909] T a3be7c7a52ee4569be546ee48caaa4ff P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting consensus metadata
I20240503 01:59:49.645921 13349 consensus_queue.cc:237] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:49.646950 6727 catalog_manager.cc:4830] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): tablet a3be7c7a52ee4569be546ee48caaa4ff (table rebalancer_dynamic_table_new_name [id=c587e31d48724de8bf73191bfe2ae533]) successfully deleted
I20240503 01:59:49.649260 6740 catalog_manager.cc:5484] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:49.709789 6739 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:49.718657 9605 tablet_service.cc:1467] Processing CreateTablet for tablet 699fdf162b6749ac84d6574c6f6d919c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=757fe9b7b5b448d4b8c6f42d886e2b26]), partition=
I20240503 01:59:49.718703 9473 tablet_service.cc:1467] Processing CreateTablet for tablet 699fdf162b6749ac84d6574c6f6d919c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=757fe9b7b5b448d4b8c6f42d886e2b26]), partition=
I20240503 01:59:49.719053 9605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 699fdf162b6749ac84d6574c6f6d919c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.719189 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 699fdf162b6749ac84d6574c6f6d919c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.721211 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 699fdf162b6749ac84d6574c6f6d919c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=757fe9b7b5b448d4b8c6f42d886e2b26]), partition=
I20240503 01:59:49.721494 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 699fdf162b6749ac84d6574c6f6d919c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:49.723599 14146 tablet_bootstrap.cc:492] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:49.723646 14148 tablet_bootstrap.cc:492] T 699fdf162b6749ac84d6574c6f6d919c P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:49.724658 14148 tablet_bootstrap.cc:654] T 699fdf162b6749ac84d6574c6f6d919c P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:49.725368 14146 tablet_bootstrap.cc:654] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:49.728050 14207 tablet_bootstrap.cc:492] T 699fdf162b6749ac84d6574c6f6d919c P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:49.729421 14207 tablet_bootstrap.cc:654] T 699fdf162b6749ac84d6574c6f6d919c P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:49.730808 14207 tablet_bootstrap.cc:492] T 699fdf162b6749ac84d6574c6f6d919c P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:49.730943 14207 ts_tablet_manager.cc:1367] T 699fdf162b6749ac84d6574c6f6d919c P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:49.731909 14207 raft_consensus.cc:348] T 699fdf162b6749ac84d6574c6f6d919c P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:49.732190 14207 raft_consensus.cc:374] T 699fdf162b6749ac84d6574c6f6d919c P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:49.732265 14207 raft_consensus.cc:729] T 699fdf162b6749ac84d6574c6f6d919c P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:49.735272 14146 tablet_bootstrap.cc:492] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:49.735272 14148 tablet_bootstrap.cc:492] T 699fdf162b6749ac84d6574c6f6d919c P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:49.735400 14146 ts_tablet_manager.cc:1367] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 01:59:49.735400 14148 ts_tablet_manager.cc:1367] T 699fdf162b6749ac84d6574c6f6d919c P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20240503 01:59:49.735931 14148 raft_consensus.cc:348] T 699fdf162b6749ac84d6574c6f6d919c P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:49.735931 14146 raft_consensus.cc:348] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:49.736059 14148 raft_consensus.cc:374] T 699fdf162b6749ac84d6574c6f6d919c P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:49.736059 14146 raft_consensus.cc:374] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:49.736099 14148 raft_consensus.cc:729] T 699fdf162b6749ac84d6574c6f6d919c P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:49.736122 14146 raft_consensus.cc:729] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:49.736231 14148 consensus_queue.cc:260] T 699fdf162b6749ac84d6574c6f6d919c P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:49.736235 14146 consensus_queue.cc:260] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:49.736737 14148 ts_tablet_manager.cc:1398] T 699fdf162b6749ac84d6574c6f6d919c P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 01:59:49.736734 14146 ts_tablet_manager.cc:1398] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:49.741999 14207 consensus_queue.cc:260] T 699fdf162b6749ac84d6574c6f6d919c P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:49.745257 14207 ts_tablet_manager.cc:1398] T 699fdf162b6749ac84d6574c6f6d919c P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240503 01:59:49.795497 13637 raft_consensus.cc:482] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:49.795672 13637 raft_consensus.cc:504] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:49.796212 13637 leader_election.cc:288] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:49.796592 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2d66da6d2e24531aee4aaa6740a7af4" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:49.796757 6882 raft_consensus.cc:2454] T f2d66da6d2e24531aee4aaa6740a7af4 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 01:59:49.796718 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2d66da6d2e24531aee4aaa6740a7af4" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:49.796878 9493 raft_consensus.cc:2454] T f2d66da6d2e24531aee4aaa6740a7af4 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 01:59:49.797192 9690 leader_election.cc:302] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:49.797381 13637 raft_consensus.cc:2781] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:49.797439 13637 raft_consensus.cc:482] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:49.797508 13637 raft_consensus.cc:3037] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.799111 13637 raft_consensus.cc:504] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:49.799566 13637 leader_election.cc:288] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Requested vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:49.799937 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2d66da6d2e24531aee4aaa6740a7af4" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:49.799991 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2d66da6d2e24531aee4aaa6740a7af4" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:49.800065 6882 raft_consensus.cc:3037] T f2d66da6d2e24531aee4aaa6740a7af4 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.800124 9493 raft_consensus.cc:3037] T f2d66da6d2e24531aee4aaa6740a7af4 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.801306 9493 raft_consensus.cc:2454] T f2d66da6d2e24531aee4aaa6740a7af4 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:49.801692 6882 raft_consensus.cc:2454] T f2d66da6d2e24531aee4aaa6740a7af4 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:49.801939 9690 leader_election.cc:302] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:49.802182 14147 raft_consensus.cc:2781] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:49.802279 14147 raft_consensus.cc:686] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Becoming Leader. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Running, Role: LEADER
I20240503 01:59:49.802420 14147 consensus_queue.cc:237] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:49.803927 14147 raft_consensus.cc:482] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:49.804029 14147 raft_consensus.cc:504] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:49.804407 14147 leader_election.cc:288] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:49.804773 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c97333c56ef4c72ac656d4a0a487c80" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:49.804924 7143 raft_consensus.cc:2454] T 8c97333c56ef4c72ac656d4a0a487c80 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 01:59:49.805114 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c97333c56ef4c72ac656d4a0a487c80" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:49.805287 9493 raft_consensus.cc:2454] T 8c97333c56ef4c72ac656d4a0a487c80 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 01:59:49.805276 6734 catalog_manager.cc:5484] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: term changed from 0 to 1, leader changed from <none> to db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:49.805490 9692 leader_election.cc:302] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:49.805645 14147 raft_consensus.cc:2781] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:49.805699 14147 raft_consensus.cc:482] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:49.805744 14147 raft_consensus.cc:3037] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.806893 14147 raft_consensus.cc:504] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:49.807296 14147 leader_election.cc:288] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Requested vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:49.807708 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c97333c56ef4c72ac656d4a0a487c80" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:49.807824 7143 raft_consensus.cc:3037] T 8c97333c56ef4c72ac656d4a0a487c80 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.808627 7143 raft_consensus.cc:2454] T 8c97333c56ef4c72ac656d4a0a487c80 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:49.808985 9692 leader_election.cc:302] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:49.809194 14147 raft_consensus.cc:2781] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:49.809185 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c97333c56ef4c72ac656d4a0a487c80" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:49.809301 14147 raft_consensus.cc:686] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Becoming Leader. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Running, Role: LEADER
I20240503 01:59:49.809311 9493 raft_consensus.cc:3037] T 8c97333c56ef4c72ac656d4a0a487c80 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.809440 14147 consensus_queue.cc:237] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:49.810305 9493 raft_consensus.cc:2454] T 8c97333c56ef4c72ac656d4a0a487c80 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:49.811812 6740 catalog_manager.cc:5484] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: term changed from 0 to 1, leader changed from <none> to db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:49.839751 6861 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:49.841526 7256 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:49.846930 13457 raft_consensus.cc:482] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:49.847093 13457 raft_consensus.cc:504] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:49.847609 13457 leader_election.cc:288] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:49.848093 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699fdf162b6749ac84d6574c6f6d919c" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:49.848242 9493 raft_consensus.cc:2454] T 699fdf162b6749ac84d6574c6f6d919c P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:49.848645 9296 leader_election.cc:302] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:49.848825 13457 raft_consensus.cc:2781] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:49.848884 13457 raft_consensus.cc:482] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:49.848920 13457 raft_consensus.cc:3037] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.850368 13457 raft_consensus.cc:504] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:49.850821 13457 leader_election.cc:288] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:49.850798 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699fdf162b6749ac84d6574c6f6d919c" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:49.850943 9625 raft_consensus.cc:2454] T 699fdf162b6749ac84d6574c6f6d919c P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:49.851194 9605 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:49.852900 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699fdf162b6749ac84d6574c6f6d919c" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:49.853042 9493 raft_consensus.cc:3037] T 699fdf162b6749ac84d6574c6f6d919c P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.853370 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "699fdf162b6749ac84d6574c6f6d919c" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:49.853510 9625 raft_consensus.cc:3037] T 699fdf162b6749ac84d6574c6f6d919c P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:49.854267 9493 raft_consensus.cc:2454] T 699fdf162b6749ac84d6574c6f6d919c P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:49.854616 9625 raft_consensus.cc:2454] T 699fdf162b6749ac84d6574c6f6d919c P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:49.854748 9296 leader_election.cc:302] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:49.854825 9737 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:49.855304 13457 raft_consensus.cc:2781] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:49.856015 6993 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:49.856532 9342 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:49.855804 13457 raft_consensus.cc:686] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Running, Role: LEADER
I20240503 01:59:49.858328 13457 consensus_queue.cc:237] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:49.863176 6740 catalog_manager.cc:5484] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:49.873392 7124 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:49.882561 9473 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:49.964532 7014 consensus_queue.cc:237] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:49.965973 9362 raft_consensus.cc:1261] T 85d3e89fac21402387aa812f754cf5bc P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:49.965986 7143 raft_consensus.cc:1261] T 85d3e89fac21402387aa812f754cf5bc P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:49.966462 12321 consensus_queue.cc:1035] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:49.966717 14004 consensus_queue.cc:1035] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:49.968711 12321 raft_consensus.cc:2932] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:49.969074 9362 raft_consensus.cc:2932] T 85d3e89fac21402387aa812f754cf5bc P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:49.969179 7143 raft_consensus.cc:2932] T 85d3e89fac21402387aa812f754cf5bc P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:49.973070 6740 catalog_manager.cc:5484] T 85d3e89fac21402387aa812f754cf5bc P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:49.977188 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet 85d3e89fac21402387aa812f754cf5bc with delete_type TABLET_DATA_TOMBSTONED (TS 5dcc33cc621a4cbbae155f3230fb7c46 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:49.978200 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:49.980015 14249 tablet_replica.cc:330] T 85d3e89fac21402387aa812f754cf5bc P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:49.980325 14249 raft_consensus.cc:2229] T 85d3e89fac21402387aa812f754cf5bc P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:49.980494 14249 raft_consensus.cc:2258] T 85d3e89fac21402387aa812f754cf5bc P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:49.981060 14249 ts_tablet_manager.cc:1875] T 85d3e89fac21402387aa812f754cf5bc P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:49.983355 9493 raft_consensus.cc:1261] T 699fdf162b6749ac84d6574c6f6d919c P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:49.983772 9625 raft_consensus.cc:1261] T 699fdf162b6749ac84d6574c6f6d919c P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:49.983937 13457 consensus_queue.cc:1035] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:49.984187 13349 consensus_queue.cc:1035] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:49.985575 14249 ts_tablet_manager.cc:1888] T 85d3e89fac21402387aa812f754cf5bc P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 01:59:49.985845 14249 log.cc:1196] T 85d3e89fac21402387aa812f754cf5bc P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/85d3e89fac21402387aa812f754cf5bc
I20240503 01:59:49.986708 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 85d3e89fac21402387aa812f754cf5bc (table rebalance_test_table_13 [id=06feff591c384a50825820a47b49c8f9]) successfully deleted
I20240503 01:59:49.987555 6734 catalog_manager.cc:5576] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087) does not have the latest schema for tablet 699fdf162b6749ac84d6574c6f6d919c (table rebalancer_dynamic_table [id=757fe9b7b5b448d4b8c6f42d886e2b26]). Expected version 1 got 0
I20240503 01:59:49.989699 6734 catalog_manager.cc:5576] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471) does not have the latest schema for tablet 699fdf162b6749ac84d6574c6f6d919c (table rebalancer_dynamic_table [id=757fe9b7b5b448d4b8c6f42d886e2b26]). Expected version 1 got 0
I20240503 01:59:49.989810 14261 tablet.cc:1722] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:49.990170 14261 tablet.cc:1620] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 01:59:49.990806 14262 tablet.cc:1722] T 699fdf162b6749ac84d6574c6f6d919c P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:49.991112 14262 tablet.cc:1620] T 699fdf162b6749ac84d6574c6f6d919c P b7062fbbc2b54fff9e99ec4ea5562c76: MemRowSet was empty: no flush needed.
I20240503 01:59:49.992265 6732 catalog_manager.cc:5576] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087) does not have the latest schema for tablet 699fdf162b6749ac84d6574c6f6d919c (table rebalancer_dynamic_table [id=757fe9b7b5b448d4b8c6f42d886e2b26]). Expected version 1 got 0
I20240503 01:59:49.999972 6732 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=757fe9b7b5b448d4b8c6f42d886e2b26] alter complete (version 1)
I20240503 01:59:50.001132 14258 mvcc.cc:205] Tried to move back new op lower bound from 7023417712565755904 to 7023417712062816256. Current Snapshot: MvccSnapshot[applied={T|T < 7023417712565755904}]
I20240503 01:59:50.001840 14263 tablet.cc:1722] T 699fdf162b6749ac84d6574c6f6d919c P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:50.002285 14263 tablet.cc:1620] T 699fdf162b6749ac84d6574c6f6d919c P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:50.003269 6734 catalog_manager.cc:5576] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471) does not have the latest schema for tablet 699fdf162b6749ac84d6574c6f6d919c (table rebalancer_dynamic_table [id=757fe9b7b5b448d4b8c6f42d886e2b26]). Expected version 1 got 0
I20240503 01:59:50.108000 13457 consensus_queue.cc:1035] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:50.109887 6740 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:50.117291 13349 consensus_queue.cc:1035] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:50.127609 14261 tablet.cc:1722] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:50.128046 14261 tablet.cc:1620] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 01:59:50.128629 14262 tablet.cc:1722] T 699fdf162b6749ac84d6574c6f6d919c P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:50.128994 14262 tablet.cc:1620] T 699fdf162b6749ac84d6574c6f6d919c P b7062fbbc2b54fff9e99ec4ea5562c76: MemRowSet was empty: no flush needed.
I20240503 01:59:50.134829 14263 tablet.cc:1722] T 699fdf162b6749ac84d6574c6f6d919c P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:50.135275 14263 tablet.cc:1620] T 699fdf162b6749ac84d6574c6f6d919c P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:50.141191 6741 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=757fe9b7b5b448d4b8c6f42d886e2b26] alter complete (version 2)
I20240503 01:59:50.158440 7256 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:50.169699 7124 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:50.170481 6993 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:50.173573 9605 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:50.175168 9473 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 01:59:50.182693 9737 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:50.189956 9342 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:50.194752 6861 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:50.224299 14147 consensus_queue.cc:1035] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:50.233207 13637 consensus_queue.cc:1035] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:50.244238 6732 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:50.251629 14261 tablet.cc:1722] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:50.253382 14263 tablet.cc:1722] T 699fdf162b6749ac84d6574c6f6d919c P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:50.254421 14262 tablet.cc:1722] T 699fdf162b6749ac84d6574c6f6d919c P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:50.260715 6740 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=757fe9b7b5b448d4b8c6f42d886e2b26] alter complete (version 3)
I20240503 01:59:50.285737 7143 consensus_queue.cc:237] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:50.287776 7275 raft_consensus.cc:1261] T 76e8697ac8624e03a46bddae35f4c9fa P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:50.288381 9493 raft_consensus.cc:1261] T 76e8697ac8624e03a46bddae35f4c9fa P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:50.288880 14330 consensus_queue.cc:1035] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:50.289382 13734 consensus_queue.cc:1035] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:50.291853 13734 raft_consensus.cc:2932] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:50.292318 9493 raft_consensus.cc:2932] T 76e8697ac8624e03a46bddae35f4c9fa P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:50.292318 7275 raft_consensus.cc:2932] T 76e8697ac8624e03a46bddae35f4c9fa P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:50.295245 6732 catalog_manager.cc:5484] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index 4 to 5, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:50.302834 6993 tablet_service.cc:1514] Processing DeleteTablet for tablet 76e8697ac8624e03a46bddae35f4c9fa with delete_type TABLET_DATA_TOMBSTONED (TS 63331f455be049c1bdd6c23c59ac1bef not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:50.303341 14339 tablet_replica.cc:330] T 76e8697ac8624e03a46bddae35f4c9fa P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:50.303543 14339 raft_consensus.cc:2229] T 76e8697ac8624e03a46bddae35f4c9fa P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:50.303791 14339 raft_consensus.cc:2258] T 76e8697ac8624e03a46bddae35f4c9fa P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:50.304476 14339 ts_tablet_manager.cc:1875] T 76e8697ac8624e03a46bddae35f4c9fa P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:50.307405 14339 ts_tablet_manager.cc:1888] T 76e8697ac8624e03a46bddae35f4c9fa P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 01:59:50.307492 14339 log.cc:1196] T 76e8697ac8624e03a46bddae35f4c9fa P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/76e8697ac8624e03a46bddae35f4c9fa
I20240503 01:59:50.308111 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet 76e8697ac8624e03a46bddae35f4c9fa (table rebalance_test_table_7 [id=3428bac98e0c49908450b3b51cb9d18c]) successfully deleted
I20240503 01:59:50.321398 6740 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:50.330915 9737 tablet_service.cc:1467] Processing CreateTablet for tablet 49eabbb8972840759ffa83951cb0b3b7 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=cb5e07c45e8b46888678a15d79f2cfd9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:50.331261 9737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49eabbb8972840759ffa83951cb0b3b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:50.332434 9605 tablet_service.cc:1467] Processing CreateTablet for tablet 49eabbb8972840759ffa83951cb0b3b7 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=cb5e07c45e8b46888678a15d79f2cfd9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:50.332759 9605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49eabbb8972840759ffa83951cb0b3b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:50.335091 9341 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd0378b2cff42a681361930f487d834 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=cb5e07c45e8b46888678a15d79f2cfd9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:50.335172 9342 tablet_service.cc:1467] Processing CreateTablet for tablet f20b6a14b8674e6fafefe20ca326d38b (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=cb5e07c45e8b46888678a15d79f2cfd9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:50.335387 9341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd0378b2cff42a681361930f487d834. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:50.335700 9735 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd0378b2cff42a681361930f487d834 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=cb5e07c45e8b46888678a15d79f2cfd9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:50.335952 9735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd0378b2cff42a681361930f487d834. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:50.338045 9736 tablet_service.cc:1467] Processing CreateTablet for tablet f20b6a14b8674e6fafefe20ca326d38b (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=cb5e07c45e8b46888678a15d79f2cfd9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:50.338289 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f20b6a14b8674e6fafefe20ca326d38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:50.338721 9473 tablet_service.cc:1467] Processing CreateTablet for tablet 49eabbb8972840759ffa83951cb0b3b7 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=cb5e07c45e8b46888678a15d79f2cfd9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:50.338822 9474 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd0378b2cff42a681361930f487d834 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=cb5e07c45e8b46888678a15d79f2cfd9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:50.339004 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49eabbb8972840759ffa83951cb0b3b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:50.339856 9474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd0378b2cff42a681361930f487d834. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:50.338822 9472 tablet_service.cc:1467] Processing CreateTablet for tablet f20b6a14b8674e6fafefe20ca326d38b (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=cb5e07c45e8b46888678a15d79f2cfd9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:50.341284 9472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f20b6a14b8674e6fafefe20ca326d38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:50.343087 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f20b6a14b8674e6fafefe20ca326d38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:50.344944 14347 tablet_bootstrap.cc:492] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:50.345369 14147 consensus_queue.cc:1035] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:50.346452 14347 tablet_bootstrap.cc:654] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:50.346794 14345 tablet_bootstrap.cc:492] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:50.348091 14345 tablet_bootstrap.cc:654] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:50.355468 14348 tablet_bootstrap.cc:492] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:50.356004 14345 tablet_bootstrap.cc:492] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:50.356192 14345 ts_tablet_manager.cc:1367] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 01:59:50.356930 13637 consensus_queue.cc:1035] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:50.357213 14347 tablet_bootstrap.cc:492] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:50.357352 14347 ts_tablet_manager.cc:1367] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240503 01:59:50.357013 14345 raft_consensus.cc:348] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:50.357579 14345 raft_consensus.cc:374] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:50.357664 14345 raft_consensus.cc:729] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:50.358069 14345 consensus_queue.cc:260] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:50.358079 14347 raft_consensus.cc:348] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:50.358217 14347 raft_consensus.cc:374] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:50.358251 14347 raft_consensus.cc:729] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:50.358429 14347 consensus_queue.cc:260] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:50.359376 14345 ts_tablet_manager.cc:1398] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:50.359627 14347 ts_tablet_manager.cc:1398] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:50.359668 14345 tablet_bootstrap.cc:492] T 6fd0378b2cff42a681361930f487d834 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:50.359865 14347 tablet_bootstrap.cc:492] T f20b6a14b8674e6fafefe20ca326d38b P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:50.361706 14346 tablet_bootstrap.cc:492] T 49eabbb8972840759ffa83951cb0b3b7 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:50.362852 14348 tablet_bootstrap.cc:654] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:50.362881 6732 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:50.363062 14345 tablet_bootstrap.cc:654] T 6fd0378b2cff42a681361930f487d834 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:50.363103 6732 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:50.364514 14345 tablet_bootstrap.cc:492] T 6fd0378b2cff42a681361930f487d834 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:50.364624 14345 ts_tablet_manager.cc:1367] T 6fd0378b2cff42a681361930f487d834 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 01:59:50.364982 14346 tablet_bootstrap.cc:654] T 49eabbb8972840759ffa83951cb0b3b7 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:50.365244 14345 raft_consensus.cc:348] T 6fd0378b2cff42a681361930f487d834 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:50.365396 14345 raft_consensus.cc:374] T 6fd0378b2cff42a681361930f487d834 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:50.365432 14345 raft_consensus.cc:729] T 6fd0378b2cff42a681361930f487d834 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:50.365576 14345 consensus_queue.cc:260] T 6fd0378b2cff42a681361930f487d834 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:50.365938 14345 ts_tablet_manager.cc:1398] T 6fd0378b2cff42a681361930f487d834 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:50.366129 14345 tablet_bootstrap.cc:492] T f20b6a14b8674e6fafefe20ca326d38b P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:50.367091 14345 tablet_bootstrap.cc:654] T f20b6a14b8674e6fafefe20ca326d38b P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:50.367985 6732 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 699fdf162b6749ac84d6574c6f6d919c
I20240503 01:59:50.368886 9605 tablet_service.cc:1514] Processing DeleteTablet for tablet 699fdf162b6749ac84d6574c6f6d919c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:50 UTC) from {username='slave'} at 127.0.0.1:58562
I20240503 01:59:50.369216 14348 tablet_bootstrap.cc:492] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:50.369310 14348 ts_tablet_manager.cc:1367] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 01:59:50.369530 9342 tablet_service.cc:1514] Processing DeleteTablet for tablet 699fdf162b6749ac84d6574c6f6d919c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:50 UTC) from {username='slave'} at 127.0.0.1:36040
I20240503 01:59:50.370208 14348 raft_consensus.cc:348] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:50.370335 14348 raft_consensus.cc:374] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:50.370373 14348 raft_consensus.cc:729] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:50.370448 9472 tablet_service.cc:1514] Processing DeleteTablet for tablet 699fdf162b6749ac84d6574c6f6d919c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:50 UTC) from {username='slave'} at 127.0.0.1:54678
I20240503 01:59:50.370510 14348 consensus_queue.cc:260] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:50.370729 13671 raft_consensus.cc:482] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:50.370802 13671 raft_consensus.cc:504] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:50.371095 14347 tablet_bootstrap.cc:654] T f20b6a14b8674e6fafefe20ca326d38b P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:50.371531 13671 leader_election.cc:288] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:50.371878 14348 ts_tablet_manager.cc:1398] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:59:50.372112 14348 tablet_bootstrap.cc:492] T 6fd0378b2cff42a681361930f487d834 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:50.373337 14348 tablet_bootstrap.cc:654] T 6fd0378b2cff42a681361930f487d834 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:50.374171 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49eabbb8972840759ffa83951cb0b3b7" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:50.374338 9757 raft_consensus.cc:2454] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:50.374755 14346 tablet_bootstrap.cc:492] T 49eabbb8972840759ffa83951cb0b3b7 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:50.374923 14346 ts_tablet_manager.cc:1367] T 49eabbb8972840759ffa83951cb0b3b7 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 01:59:50.374990 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49eabbb8972840759ffa83951cb0b3b7" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:50.375301 9428 leader_election.cc:302] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:50.375560 13349 raft_consensus.cc:482] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:50.375676 13349 raft_consensus.cc:504] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:50.375762 14346 raft_consensus.cc:348] T 49eabbb8972840759ffa83951cb0b3b7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:50.375982 14346 raft_consensus.cc:374] T 49eabbb8972840759ffa83951cb0b3b7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:50.376066 14346 raft_consensus.cc:729] T 49eabbb8972840759ffa83951cb0b3b7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:50.376273 14346 consensus_queue.cc:260] T 49eabbb8972840759ffa83951cb0b3b7 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:50.376467 14365 tablet_replica.cc:330] T 699fdf162b6749ac84d6574c6f6d919c P b7062fbbc2b54fff9e99ec4ea5562c76: stopping tablet replica
I20240503 01:59:50.376485 13349 leader_election.cc:288] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:50.376727 14346 ts_tablet_manager.cc:1398] T 49eabbb8972840759ffa83951cb0b3b7 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:50.376878 14365 raft_consensus.cc:2229] T 699fdf162b6749ac84d6574c6f6d919c P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:50.377434 14345 tablet_bootstrap.cc:492] T f20b6a14b8674e6fafefe20ca326d38b P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:50.377576 14345 ts_tablet_manager.cc:1367] T f20b6a14b8674e6fafefe20ca326d38b P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20240503 01:59:50.378151 14345 raft_consensus.cc:348] T f20b6a14b8674e6fafefe20ca326d38b P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:50.378281 14345 raft_consensus.cc:374] T f20b6a14b8674e6fafefe20ca326d38b P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:50.378309 14345 raft_consensus.cc:729] T f20b6a14b8674e6fafefe20ca326d38b P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:50.378417 14345 consensus_queue.cc:260] T f20b6a14b8674e6fafefe20ca326d38b P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:50.379174 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd0378b2cff42a681361930f487d834" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:50.379315 9757 raft_consensus.cc:2454] T 6fd0378b2cff42a681361930f487d834 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:50.379909 14366 tablet_replica.cc:330] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8: stopping tablet replica
I20240503 01:59:50.380100 9297 leader_election.cc:302] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:50.380061 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd0378b2cff42a681361930f487d834" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:50.380373 13349 raft_consensus.cc:2781] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:50.380434 13349 raft_consensus.cc:482] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:50.380470 13349 raft_consensus.cc:3037] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
W20240503 01:59:50.380688 9296 leader_election.cc:341] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Illegal state: must be running to vote when last-logged opid is not known
I20240503 01:59:50.381635 13349 raft_consensus.cc:504] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:50.382064 13349 leader_election.cc:288] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Requested vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:50.382078 14365 raft_consensus.cc:2258] T 699fdf162b6749ac84d6574c6f6d919c P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:50.377712 9625 raft_consensus.cc:2454] T 49eabbb8972840759ffa83951cb0b3b7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:50.383054 14365 ts_tablet_manager.cc:1875] T 699fdf162b6749ac84d6574c6f6d919c P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:50.379336 14345 ts_tablet_manager.cc:1398] T f20b6a14b8674e6fafefe20ca326d38b P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:50.383930 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd0378b2cff42a681361930f487d834" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:50.384148 14359 raft_consensus.cc:2781] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:50.384194 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd0378b2cff42a681361930f487d834" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:50.384255 14359 raft_consensus.cc:482] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:50.384310 14359 raft_consensus.cc:3037] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:50.384315 9757 raft_consensus.cc:3037] T 6fd0378b2cff42a681361930f487d834 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
W20240503 01:59:50.384439 9296 leader_election.cc:341] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Illegal state: must be running to vote when last-logged opid is not known
I20240503 01:59:50.385437 9757 raft_consensus.cc:2454] T 6fd0378b2cff42a681361930f487d834 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:50.385742 14359 raft_consensus.cc:504] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:50.380093 14366 raft_consensus.cc:2229] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:50.385902 9297 leader_election.cc:302] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, db1a0000aaa346f79c0aa4b6c5105191; no voters: bb5364d730cb4489a0bfc6c270b7450a
I20240503 01:59:50.386301 14366 raft_consensus.cc:2258] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:50.387195 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49eabbb8972840759ffa83951cb0b3b7" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:50.387321 9757 raft_consensus.cc:3037] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:50.387952 14367 tablet_replica.cc:330] T 699fdf162b6749ac84d6574c6f6d919c P bb5364d730cb4489a0bfc6c270b7450a: stopping tablet replica
I20240503 01:59:50.388118 14367 raft_consensus.cc:2229] T 699fdf162b6749ac84d6574c6f6d919c P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:50.388239 14367 raft_consensus.cc:2258] T 699fdf162b6749ac84d6574c6f6d919c P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:50.388391 9757 raft_consensus.cc:2454] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:50.388706 9428 leader_election.cc:302] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:50.389046 14348 tablet_bootstrap.cc:492] T 6fd0378b2cff42a681361930f487d834 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:50.389081 13671 raft_consensus.cc:2781] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:50.389137 14348 ts_tablet_manager.cc:1367] T 6fd0378b2cff42a681361930f487d834 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.001s
I20240503 01:59:50.389246 13671 raft_consensus.cc:686] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Becoming Leader. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Running, Role: LEADER
I20240503 01:59:50.389381 13671 consensus_queue.cc:237] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:50.389700 14348 raft_consensus.cc:348] T 6fd0378b2cff42a681361930f487d834 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:50.389797 14348 raft_consensus.cc:374] T 6fd0378b2cff42a681361930f487d834 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:50.389829 14348 raft_consensus.cc:729] T 6fd0378b2cff42a681361930f487d834 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:50.386688 13349 raft_consensus.cc:2781] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:50.389955 14367 ts_tablet_manager.cc:1875] T 699fdf162b6749ac84d6574c6f6d919c P bb5364d730cb4489a0bfc6c270b7450a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:50.390071 13349 raft_consensus.cc:686] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Running, Role: LEADER
I20240503 01:59:50.390244 13349 consensus_queue.cc:237] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:50.390760 14359 leader_election.cc:288] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Requested vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:50.392078 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49eabbb8972840759ffa83951cb0b3b7" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:50.392215 9625 raft_consensus.cc:3037] T 49eabbb8972840759ffa83951cb0b3b7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:50.392053 6732 catalog_manager.cc:5484] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: term changed from 0 to 1, leader changed from <none> to bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:50.393122 9625 raft_consensus.cc:2454] T 49eabbb8972840759ffa83951cb0b3b7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:50.389955 14348 consensus_queue.cc:260] T 6fd0378b2cff42a681361930f487d834 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:50.395776 14348 ts_tablet_manager.cc:1398] T 6fd0378b2cff42a681361930f487d834 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 01:59:50.396014 14348 tablet_bootstrap.cc:492] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:50.397894 14348 tablet_bootstrap.cc:654] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:50.400938 6732 catalog_manager.cc:5484] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:50.401556 14366 ts_tablet_manager.cc:1875] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:50.402377 14365 ts_tablet_manager.cc:1888] T 699fdf162b6749ac84d6574c6f6d919c P b7062fbbc2b54fff9e99ec4ea5562c76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:50.402506 14365 log.cc:1196] T 699fdf162b6749ac84d6574c6f6d919c P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/699fdf162b6749ac84d6574c6f6d919c
I20240503 01:59:50.403057 14365 ts_tablet_manager.cc:1909] T 699fdf162b6749ac84d6574c6f6d919c P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting consensus metadata
I20240503 01:59:50.403982 6727 catalog_manager.cc:4830] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): tablet 699fdf162b6749ac84d6574c6f6d919c (table rebalancer_dynamic_table_new_name [id=757fe9b7b5b448d4b8c6f42d886e2b26]) successfully deleted
I20240503 01:59:50.404500 14347 tablet_bootstrap.cc:492] T f20b6a14b8674e6fafefe20ca326d38b P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:50.404611 14347 ts_tablet_manager.cc:1367] T f20b6a14b8674e6fafefe20ca326d38b P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.000s
I20240503 01:59:50.405115 14367 ts_tablet_manager.cc:1888] T 699fdf162b6749ac84d6574c6f6d919c P bb5364d730cb4489a0bfc6c270b7450a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:50.405177 14347 raft_consensus.cc:348] T f20b6a14b8674e6fafefe20ca326d38b P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:50.405331 14347 raft_consensus.cc:374] T f20b6a14b8674e6fafefe20ca326d38b P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:50.405385 14347 raft_consensus.cc:729] T f20b6a14b8674e6fafefe20ca326d38b P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:50.405515 14347 consensus_queue.cc:260] T f20b6a14b8674e6fafefe20ca326d38b P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:50.405838 14347 ts_tablet_manager.cc:1398] T f20b6a14b8674e6fafefe20ca326d38b P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:50.406598 14367 log.cc:1196] T 699fdf162b6749ac84d6574c6f6d919c P bb5364d730cb4489a0bfc6c270b7450a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/699fdf162b6749ac84d6574c6f6d919c
I20240503 01:59:50.406885 14366 ts_tablet_manager.cc:1888] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:50.406987 14367 ts_tablet_manager.cc:1909] T 699fdf162b6749ac84d6574c6f6d919c P bb5364d730cb4489a0bfc6c270b7450a: Deleting consensus metadata
I20240503 01:59:50.408257 6725 catalog_manager.cc:4830] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): tablet 699fdf162b6749ac84d6574c6f6d919c (table rebalancer_dynamic_table_new_name [id=757fe9b7b5b448d4b8c6f42d886e2b26]) successfully deleted
I20240503 01:59:50.408934 14348 tablet_bootstrap.cc:492] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:50.409046 14348 ts_tablet_manager.cc:1367] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.003s
I20240503 01:59:50.409271 14366 log.cc:1196] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/699fdf162b6749ac84d6574c6f6d919c
I20240503 01:59:50.410224 14366 ts_tablet_manager.cc:1909] T 699fdf162b6749ac84d6574c6f6d919c P 12e8c683db054df3b75c6285b98a5ad8: Deleting consensus metadata
I20240503 01:59:50.410364 14348 raft_consensus.cc:348] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:50.410668 14348 raft_consensus.cc:374] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:50.411290 6727 catalog_manager.cc:4830] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): tablet 699fdf162b6749ac84d6574c6f6d919c (table rebalancer_dynamic_table_new_name [id=757fe9b7b5b448d4b8c6f42d886e2b26]) successfully deleted
I20240503 01:59:50.412318 14348 raft_consensus.cc:729] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:50.412515 14348 consensus_queue.cc:260] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:50.412907 14348 ts_tablet_manager.cc:1398] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 01:59:50.473488 6732 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:50.482371 14375 raft_consensus.cc:482] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:50.482554 14375 raft_consensus.cc:504] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:50.483059 14375 leader_election.cc:288] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:50.487661 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20b6a14b8674e6fafefe20ca326d38b" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:50.487979 9757 raft_consensus.cc:2454] T f20b6a14b8674e6fafefe20ca326d38b P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:50.488483 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20b6a14b8674e6fafefe20ca326d38b" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:50.488521 9428 leader_election.cc:302] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:50.488660 9363 raft_consensus.cc:2454] T f20b6a14b8674e6fafefe20ca326d38b P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:50.489723 14375 raft_consensus.cc:2781] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:50.489804 14375 raft_consensus.cc:482] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:50.489848 14375 raft_consensus.cc:3037] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:50.491531 14375 raft_consensus.cc:504] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:50.492038 14375 leader_election.cc:288] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Requested vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:50.492391 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20b6a14b8674e6fafefe20ca326d38b" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:50.492561 9363 raft_consensus.cc:3037] T f20b6a14b8674e6fafefe20ca326d38b P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:50.492774 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20b6a14b8674e6fafefe20ca326d38b" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:50.492910 9757 raft_consensus.cc:3037] T f20b6a14b8674e6fafefe20ca326d38b P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:50.493790 9363 raft_consensus.cc:2454] T f20b6a14b8674e6fafefe20ca326d38b P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:50.493953 9757 raft_consensus.cc:2454] T f20b6a14b8674e6fafefe20ca326d38b P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:50.494325 9428 leader_election.cc:302] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:50.494849 14359 raft_consensus.cc:2781] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:50.494956 14359 raft_consensus.cc:686] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Becoming Leader. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Running, Role: LEADER
I20240503 01:59:50.495119 14359 consensus_queue.cc:237] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:50.496423 9342 tablet_service.cc:1467] Processing CreateTablet for tablet f259a5eac4354bbc94753fb59cea2a0d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c56128f83d814e079e5ffe833001a3b7]), partition=
I20240503 01:59:50.496547 7124 tablet_service.cc:1467] Processing CreateTablet for tablet f259a5eac4354bbc94753fb59cea2a0d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c56128f83d814e079e5ffe833001a3b7]), partition=
I20240503 01:59:50.496641 9736 tablet_service.cc:1467] Processing CreateTablet for tablet f259a5eac4354bbc94753fb59cea2a0d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c56128f83d814e079e5ffe833001a3b7]), partition=
I20240503 01:59:50.496706 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f259a5eac4354bbc94753fb59cea2a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:50.496906 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f259a5eac4354bbc94753fb59cea2a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:50.496919 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f259a5eac4354bbc94753fb59cea2a0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:50.498867 14347 tablet_bootstrap.cc:492] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:50.500101 14347 tablet_bootstrap.cc:654] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:50.500844 6739 catalog_manager.cc:5484] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: term changed from 0 to 1, leader changed from <none> to bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:50.501236 14347 tablet_bootstrap.cc:492] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:50.501322 14347 ts_tablet_manager.cc:1367] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:50.501818 14347 raft_consensus.cc:348] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:50.501957 14347 raft_consensus.cc:374] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:50.502007 14347 raft_consensus.cc:729] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:50.502372 14347 consensus_queue.cc:260] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:50.502346 14345 tablet_bootstrap.cc:492] T f259a5eac4354bbc94753fb59cea2a0d P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:50.503742 14345 tablet_bootstrap.cc:654] T f259a5eac4354bbc94753fb59cea2a0d P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:50.504382 14347 ts_tablet_manager.cc:1398] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:50.506019 14345 tablet_bootstrap.cc:492] T f259a5eac4354bbc94753fb59cea2a0d P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:50.506139 14345 ts_tablet_manager.cc:1367] T f259a5eac4354bbc94753fb59cea2a0d P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 01:59:50.506742 14345 raft_consensus.cc:348] T f259a5eac4354bbc94753fb59cea2a0d P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:50.506873 14345 raft_consensus.cc:374] T f259a5eac4354bbc94753fb59cea2a0d P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:50.506915 14345 raft_consensus.cc:729] T f259a5eac4354bbc94753fb59cea2a0d P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:50.507040 14345 consensus_queue.cc:260] T f259a5eac4354bbc94753fb59cea2a0d P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:50.507378 14345 ts_tablet_manager.cc:1398] T f259a5eac4354bbc94753fb59cea2a0d P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:50.521626 14411 tablet_bootstrap.cc:492] T f259a5eac4354bbc94753fb59cea2a0d P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:50.528841 14411 tablet_bootstrap.cc:654] T f259a5eac4354bbc94753fb59cea2a0d P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:50.539165 14411 tablet_bootstrap.cc:492] T f259a5eac4354bbc94753fb59cea2a0d P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:50.539301 14411 ts_tablet_manager.cc:1367] T f259a5eac4354bbc94753fb59cea2a0d P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240503 01:59:50.539872 14411 raft_consensus.cc:348] T f259a5eac4354bbc94753fb59cea2a0d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:50.540627 14411 raft_consensus.cc:374] T f259a5eac4354bbc94753fb59cea2a0d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:50.540735 14411 raft_consensus.cc:729] T f259a5eac4354bbc94753fb59cea2a0d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:50.541455 14411 consensus_queue.cc:260] T f259a5eac4354bbc94753fb59cea2a0d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:50.542340 14411 ts_tablet_manager.cc:1398] T f259a5eac4354bbc94753fb59cea2a0d P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:50.576128 9342 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:50.578840 6993 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:50.580250 7256 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:50.580508 9605 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:50.603878 9472 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:50.604300 9736 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:50.604846 6861 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:50.610994 7124 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:50.622609 13349 raft_consensus.cc:482] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:50.622767 13349 raft_consensus.cc:504] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:50.623234 13349 leader_election.cc:288] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:50.623603 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f259a5eac4354bbc94753fb59cea2a0d" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:50.623766 9757 raft_consensus.cc:2454] T f259a5eac4354bbc94753fb59cea2a0d P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:50.623966 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f259a5eac4354bbc94753fb59cea2a0d" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:50.624122 7143 raft_consensus.cc:2454] T f259a5eac4354bbc94753fb59cea2a0d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:50.624158 9297 leader_election.cc:302] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:50.624549 13349 raft_consensus.cc:2781] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:50.624615 13349 raft_consensus.cc:482] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:50.624650 13349 raft_consensus.cc:3037] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:50.625716 13349 raft_consensus.cc:504] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:50.626202 13349 leader_election.cc:288] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Requested vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:50.626631 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f259a5eac4354bbc94753fb59cea2a0d" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:50.626793 7143 raft_consensus.cc:3037] T f259a5eac4354bbc94753fb59cea2a0d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:50.626865 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f259a5eac4354bbc94753fb59cea2a0d" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:50.627058 9757 raft_consensus.cc:3037] T f259a5eac4354bbc94753fb59cea2a0d P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:50.627933 7143 raft_consensus.cc:2454] T f259a5eac4354bbc94753fb59cea2a0d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:50.628048 9757 raft_consensus.cc:2454] T f259a5eac4354bbc94753fb59cea2a0d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:50.628389 9297 leader_election.cc:302] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:50.628705 13349 raft_consensus.cc:2781] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:50.629069 13349 raft_consensus.cc:686] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Running, Role: LEADER
I20240503 01:59:50.629235 13349 consensus_queue.cc:237] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:50.636600 6732 catalog_manager.cc:5484] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:50.700350 7012 consensus_queue.cc:237] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:50.702934 6882 raft_consensus.cc:1261] T 87652406a8e442e58fe636a85ab72158 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:50.703557 9493 raft_consensus.cc:1261] T 87652406a8e442e58fe636a85ab72158 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:50.703579 14243 consensus_queue.cc:1035] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:50.703964 14004 consensus_queue.cc:1035] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:50.706494 14243 raft_consensus.cc:2932] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:50.707463 9493 raft_consensus.cc:2932] T 87652406a8e442e58fe636a85ab72158 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:50.708158 6882 raft_consensus.cc:2932] T 87652406a8e442e58fe636a85ab72158 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:50.710471 6732 catalog_manager.cc:5484] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:50.714756 7124 tablet_service.cc:1514] Processing DeleteTablet for tablet 87652406a8e442e58fe636a85ab72158 with delete_type TABLET_DATA_TOMBSTONED (TS c688f3df56ac4b6d9b0c30fa4cc8ca2a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:50.715328 14461 tablet_replica.cc:330] T 87652406a8e442e58fe636a85ab72158 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:50.715667 14461 raft_consensus.cc:2229] T 87652406a8e442e58fe636a85ab72158 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:50.715875 14461 raft_consensus.cc:2258] T 87652406a8e442e58fe636a85ab72158 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:50.716820 14461 ts_tablet_manager.cc:1875] T 87652406a8e442e58fe636a85ab72158 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:50.721185 14461 ts_tablet_manager.cc:1888] T 87652406a8e442e58fe636a85ab72158 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 01:59:50.721300 14461 log.cc:1196] T 87652406a8e442e58fe636a85ab72158 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/87652406a8e442e58fe636a85ab72158
I20240503 01:59:50.722043 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet 87652406a8e442e58fe636a85ab72158 (table rebalance_test_table_5 [id=eca57da7c1764bd9a55819692641dac8]) successfully deleted
I20240503 01:59:50.760165 6739 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:50.767655 9757 raft_consensus.cc:1261] T f259a5eac4354bbc94753fb59cea2a0d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:50.768445 7143 raft_consensus.cc:1261] T f259a5eac4354bbc94753fb59cea2a0d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:50.769006 13457 consensus_queue.cc:1035] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:50.769227 13349 consensus_queue.cc:1035] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:50.773501 6739 catalog_manager.cc:5576] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053) does not have the latest schema for tablet f259a5eac4354bbc94753fb59cea2a0d (table rebalancer_dynamic_table [id=c56128f83d814e079e5ffe833001a3b7]). Expected version 1 got 0
I20240503 01:59:50.773991 6734 catalog_manager.cc:5576] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979) does not have the latest schema for tablet f259a5eac4354bbc94753fb59cea2a0d (table rebalancer_dynamic_table [id=c56128f83d814e079e5ffe833001a3b7]). Expected version 1 got 0
I20240503 01:59:50.792425 6739 catalog_manager.cc:5576] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053) does not have the latest schema for tablet f259a5eac4354bbc94753fb59cea2a0d (table rebalancer_dynamic_table [id=c56128f83d814e079e5ffe833001a3b7]). Expected version 1 got 0
I20240503 01:59:50.793669 6739 catalog_manager.cc:5576] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979) does not have the latest schema for tablet f259a5eac4354bbc94753fb59cea2a0d (table rebalancer_dynamic_table [id=c56128f83d814e079e5ffe833001a3b7]). Expected version 1 got 0
I20240503 01:59:50.793763 14508 tablet.cc:1722] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:50.794238 14508 tablet.cc:1620] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 01:59:50.798445 14510 tablet.cc:1722] T f259a5eac4354bbc94753fb59cea2a0d P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:50.799010 14510 tablet.cc:1620] T f259a5eac4354bbc94753fb59cea2a0d P c688f3df56ac4b6d9b0c30fa4cc8ca2a: MemRowSet was empty: no flush needed.
I20240503 01:59:50.804219 6735 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=c56128f83d814e079e5ffe833001a3b7] alter complete (version 1)
I20240503 01:59:50.811369 14509 tablet.cc:1722] T f259a5eac4354bbc94753fb59cea2a0d P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:50.811816 14509 tablet.cc:1620] T f259a5eac4354bbc94753fb59cea2a0d P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:50.863368 14375 consensus_queue.cc:1035] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:50.871208 13457 consensus_queue.cc:1035] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:50.874801 14359 consensus_queue.cc:1035] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:50.876163 7256 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:50.878057 14359 raft_consensus.cc:482] T 6fd0378b2cff42a681361930f487d834 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:50.878180 14359 raft_consensus.cc:504] T 6fd0378b2cff42a681361930f487d834 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:50.878777 14359 leader_election.cc:288] T 6fd0378b2cff42a681361930f487d834 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:50.879097 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd0378b2cff42a681361930f487d834" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:50.879199 9362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd0378b2cff42a681361930f487d834" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:50.881237 9605 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:50.881368 9428 leader_election.cc:302] T 6fd0378b2cff42a681361930f487d834 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a; no voters: 12e8c683db054df3b75c6285b98a5ad8, db1a0000aaa346f79c0aa4b6c5105191
I20240503 01:59:50.881891 14359 raft_consensus.cc:2726] T 6fd0378b2cff42a681361930f487d834 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 01:59:50.884282 6861 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:50.889675 9342 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 01:59:50.893098 7124 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:50.899117 9736 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:50.906584 14359 consensus_queue.cc:1035] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:50.908949 6733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:50.919669 9493 raft_consensus.cc:3037] T 6fd0378b2cff42a681361930f487d834 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:50.919881 14508 tablet.cc:1722] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:50.920230 14508 tablet.cc:1620] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 01:59:50.922801 13349 consensus_queue.cc:1035] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:50.924966 14510 tablet.cc:1722] T f259a5eac4354bbc94753fb59cea2a0d P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:50.925367 14510 tablet.cc:1620] T f259a5eac4354bbc94753fb59cea2a0d P c688f3df56ac4b6d9b0c30fa4cc8ca2a: MemRowSet was empty: no flush needed.
I20240503 01:59:50.926970 6733 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=c56128f83d814e079e5ffe833001a3b7] alter complete (version 2)
I20240503 01:59:50.930613 14509 tablet.cc:1722] T f259a5eac4354bbc94753fb59cea2a0d P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:50.931002 14509 tablet.cc:1620] T f259a5eac4354bbc94753fb59cea2a0d P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
W20240503 01:59:50.941860 14524 log.cc:924] Time spent T 6fd0378b2cff42a681361930f487d834 P db1a0000aaa346f79c0aa4b6c5105191: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20240503 01:59:50.954919 6993 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:50.959059 13671 consensus_queue.cc:1035] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:50.972846 9472 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 01:59:51.024080 6739 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:51.029724 9757 consensus_queue.cc:237] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:51.031404 7275 raft_consensus.cc:1261] T 2f271b67a6f34eb18c98824ca4b26d17 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:51.031811 7143 raft_consensus.cc:1261] T 2f271b67a6f34eb18c98824ca4b26d17 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:51.032050 13637 consensus_queue.cc:1035] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:51.032369 14147 consensus_queue.cc:1035] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:51.034791 6739 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:51.035744 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 0bbbd17c78bc4e329a3a7c7008a34d86 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=9c1714bb582349bcb6ea24e634fcd46c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.036204 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bbbd17c78bc4e329a3a7c7008a34d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.036147 9605 tablet_service.cc:1467] Processing CreateTablet for tablet 6673a223a841415783ed8934d13b20ed (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=9c1714bb582349bcb6ea24e634fcd46c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.036440 9605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6673a223a841415783ed8934d13b20ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.036391 9341 tablet_service.cc:1467] Processing CreateTablet for tablet 442a974500b8458c951993c5b1547820 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=9c1714bb582349bcb6ea24e634fcd46c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.036749 9341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442a974500b8458c951993c5b1547820. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.039011 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 442a974500b8458c951993c5b1547820 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=9c1714bb582349bcb6ea24e634fcd46c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.039150 14549 tablet_bootstrap.cc:492] T 442a974500b8458c951993c5b1547820 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:51.039472 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442a974500b8458c951993c5b1547820. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.039934 7256 tablet_service.cc:1467] Processing CreateTablet for tablet 0bbbd17c78bc4e329a3a7c7008a34d86 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=9c1714bb582349bcb6ea24e634fcd46c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.039968 9736 tablet_service.cc:1467] Processing CreateTablet for tablet 0bbbd17c78bc4e329a3a7c7008a34d86 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=9c1714bb582349bcb6ea24e634fcd46c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.040210 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bbbd17c78bc4e329a3a7c7008a34d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.040275 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bbbd17c78bc4e329a3a7c7008a34d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.040730 14549 tablet_bootstrap.cc:654] T 442a974500b8458c951993c5b1547820 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:51.042757 14552 tablet_bootstrap.cc:492] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:51.043398 14551 tablet_bootstrap.cc:492] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:51.043473 14553 tablet_bootstrap.cc:492] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:51.043414 9472 tablet_service.cc:1467] Processing CreateTablet for tablet 6673a223a841415783ed8934d13b20ed (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=9c1714bb582349bcb6ea24e634fcd46c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.043727 9472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6673a223a841415783ed8934d13b20ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.044651 14553 tablet_bootstrap.cc:654] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:51.044723 14551 tablet_bootstrap.cc:654] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:51.046126 14553 tablet_bootstrap.cc:492] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:51.046257 14557 tablet_bootstrap.cc:492] T 6673a223a841415783ed8934d13b20ed P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:51.046227 14553 ts_tablet_manager.cc:1367] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:51.046581 6993 tablet_service.cc:1467] Processing CreateTablet for tablet 6673a223a841415783ed8934d13b20ed (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=9c1714bb582349bcb6ea24e634fcd46c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.046880 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6673a223a841415783ed8934d13b20ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.047040 14147 raft_consensus.cc:2932] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:51.047274 14553 raft_consensus.cc:348] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:51.047394 14553 raft_consensus.cc:374] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:51.047426 14553 raft_consensus.cc:729] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.047533 14553 consensus_queue.cc:260] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:51.047873 14553 ts_tablet_manager.cc:1398] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:51.048283 14557 tablet_bootstrap.cc:654] T 6673a223a841415783ed8934d13b20ed P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:51.049710 7143 raft_consensus.cc:2932] T 2f271b67a6f34eb18c98824ca4b26d17 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:51.043414 9474 tablet_service.cc:1467] Processing CreateTablet for tablet 442a974500b8458c951993c5b1547820 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=9c1714bb582349bcb6ea24e634fcd46c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.053812 9474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442a974500b8458c951993c5b1547820. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.057004 6733 catalog_manager.cc:5484] T 2f271b67a6f34eb18c98824ca4b26d17 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:51.058791 14560 tablet_bootstrap.cc:492] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:51.059317 14552 tablet_bootstrap.cc:654] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:51.059885 14560 tablet_bootstrap.cc:654] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:51.061163 14560 tablet_bootstrap.cc:492] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:51.061177 14549 tablet_bootstrap.cc:492] T 442a974500b8458c951993c5b1547820 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:51.061297 14549 ts_tablet_manager.cc:1367] T 442a974500b8458c951993c5b1547820 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240503 01:59:51.061327 14560 ts_tablet_manager.cc:1367] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:51.061843 14549 raft_consensus.cc:348] T 442a974500b8458c951993c5b1547820 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:51.061983 14549 raft_consensus.cc:374] T 442a974500b8458c951993c5b1547820 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:51.062017 14549 raft_consensus.cc:729] T 442a974500b8458c951993c5b1547820 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.062153 14549 consensus_queue.cc:260] T 442a974500b8458c951993c5b1547820 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:51.062484 14549 ts_tablet_manager.cc:1398] T 442a974500b8458c951993c5b1547820 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:51.062708 14549 tablet_bootstrap.cc:492] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:51.062919 14508 tablet.cc:1722] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:51.064429 14549 tablet_bootstrap.cc:654] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:51.064831 14509 tablet.cc:1722] T f259a5eac4354bbc94753fb59cea2a0d P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:51.067034 14554 tablet_bootstrap.cc:492] T 6673a223a841415783ed8934d13b20ed P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:51.068370 14510 tablet.cc:1722] T f259a5eac4354bbc94753fb59cea2a0d P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:51.068461 14554 tablet_bootstrap.cc:654] T 6673a223a841415783ed8934d13b20ed P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:51.070842 7275 raft_consensus.cc:2932] T 2f271b67a6f34eb18c98824ca4b26d17 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:51.074362 6734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=c56128f83d814e079e5ffe833001a3b7] alter complete (version 3)
I20240503 01:59:51.076542 9605 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f271b67a6f34eb18c98824ca4b26d17 with delete_type TABLET_DATA_TOMBSTONED (TS b7062fbbc2b54fff9e99ec4ea5562c76 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:58562
I20240503 01:59:51.077772 14557 tablet_bootstrap.cc:492] T 6673a223a841415783ed8934d13b20ed P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:51.079237 14552 tablet_bootstrap.cc:492] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:51.079238 14557 ts_tablet_manager.cc:1367] T 6673a223a841415783ed8934d13b20ed P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.000s
I20240503 01:59:51.079454 14552 ts_tablet_manager.cc:1367] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20240503 01:59:51.079790 14557 raft_consensus.cc:348] T 6673a223a841415783ed8934d13b20ed P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:51.079911 14557 raft_consensus.cc:374] T 6673a223a841415783ed8934d13b20ed P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:51.079948 14557 raft_consensus.cc:729] T 6673a223a841415783ed8934d13b20ed P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.079993 14552 raft_consensus.cc:348] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:51.080155 14552 raft_consensus.cc:374] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:51.080121 14557 consensus_queue.cc:260] T 6673a223a841415783ed8934d13b20ed P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:51.080214 14552 raft_consensus.cc:729] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.080348 14552 consensus_queue.cc:260] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:51.080476 14557 ts_tablet_manager.cc:1398] T 6673a223a841415783ed8934d13b20ed P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:51.080731 14557 tablet_bootstrap.cc:492] T 442a974500b8458c951993c5b1547820 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:51.081575 14552 ts_tablet_manager.cc:1398] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:51.081962 14560 raft_consensus.cc:348] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:51.082459 14557 tablet_bootstrap.cc:654] T 442a974500b8458c951993c5b1547820 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:51.082440 14560 raft_consensus.cc:374] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:51.082612 14560 raft_consensus.cc:729] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.082841 14560 consensus_queue.cc:260] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:51.084019 14560 ts_tablet_manager.cc:1398] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.023s user 0.001s sys 0.000s
I20240503 01:59:51.086371 14551 tablet_bootstrap.cc:492] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:51.086546 14551 ts_tablet_manager.cc:1367] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.000s
I20240503 01:59:51.086622 14571 tablet_replica.cc:330] T 2f271b67a6f34eb18c98824ca4b26d17 P b7062fbbc2b54fff9e99ec4ea5562c76: stopping tablet replica
I20240503 01:59:51.086786 14571 raft_consensus.cc:2229] T 2f271b67a6f34eb18c98824ca4b26d17 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:51.087085 14551 raft_consensus.cc:348] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:51.087390 14551 raft_consensus.cc:374] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:51.087514 14551 raft_consensus.cc:729] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.087709 14551 consensus_queue.cc:260] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:51.088148 14551 ts_tablet_manager.cc:1398] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:51.090126 14549 tablet_bootstrap.cc:492] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:51.090261 14549 ts_tablet_manager.cc:1367] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20240503 01:59:51.091308 14549 raft_consensus.cc:348] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:51.091470 14549 raft_consensus.cc:374] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:51.091500 14549 raft_consensus.cc:729] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.091661 14549 consensus_queue.cc:260] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:51.091966 14549 ts_tablet_manager.cc:1398] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:51.087121 14571 raft_consensus.cc:2258] T 2f271b67a6f34eb18c98824ca4b26d17 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:51.106974 14554 tablet_bootstrap.cc:492] T 6673a223a841415783ed8934d13b20ed P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:51.107169 14554 ts_tablet_manager.cc:1367] T 6673a223a841415783ed8934d13b20ed P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
I20240503 01:59:51.107695 14554 raft_consensus.cc:348] T 6673a223a841415783ed8934d13b20ed P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:51.107815 14554 raft_consensus.cc:374] T 6673a223a841415783ed8934d13b20ed P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:51.107842 14554 raft_consensus.cc:729] T 6673a223a841415783ed8934d13b20ed P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.107972 14554 consensus_queue.cc:260] T 6673a223a841415783ed8934d13b20ed P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:51.108239 14554 ts_tablet_manager.cc:1398] T 6673a223a841415783ed8934d13b20ed P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:51.112846 14571 ts_tablet_manager.cc:1875] T 2f271b67a6f34eb18c98824ca4b26d17 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:51.123884 14571 ts_tablet_manager.cc:1888] T 2f271b67a6f34eb18c98824ca4b26d17 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 01:59:51.124420 14571 log.cc:1196] T 2f271b67a6f34eb18c98824ca4b26d17 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/2f271b67a6f34eb18c98824ca4b26d17
I20240503 01:59:51.125102 14557 tablet_bootstrap.cc:492] T 442a974500b8458c951993c5b1547820 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:51.125339 14557 ts_tablet_manager.cc:1367] T 442a974500b8458c951993c5b1547820 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.000s
I20240503 01:59:51.126013 6727 catalog_manager.cc:4830] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): tablet 2f271b67a6f34eb18c98824ca4b26d17 (table rebalancer_extra_table_8 [id=5094b51906a7434daee33202abd81bfc]) successfully deleted
I20240503 01:59:51.126260 14557 raft_consensus.cc:348] T 442a974500b8458c951993c5b1547820 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:51.126397 14557 raft_consensus.cc:374] T 442a974500b8458c951993c5b1547820 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:51.126435 14557 raft_consensus.cc:729] T 442a974500b8458c951993c5b1547820 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.126652 14557 consensus_queue.cc:260] T 442a974500b8458c951993c5b1547820 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:51.127131 14557 ts_tablet_manager.cc:1398] T 442a974500b8458c951993c5b1547820 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:51.160514 7275 consensus_queue.cc:237] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
W20240503 01:59:51.162142 7210 consensus_peers.cc:485] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a -> Peer db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): Couldn't send request to peer db1a0000aaa346f79c0aa4b6c5105191. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae9fb6c5833145ba91bbd7c24e68a7e2. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:51.162211 7143 raft_consensus.cc:1261] T ae9fb6c5833145ba91bbd7c24e68a7e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:51.162318 6882 raft_consensus.cc:1261] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:51.162717 14599 consensus_queue.cc:1035] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:51.162947 13743 consensus_queue.cc:1035] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:51.165345 13743 raft_consensus.cc:2932] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:51.166025 6882 raft_consensus.cc:2932] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:51.167847 7143 raft_consensus.cc:2932] T ae9fb6c5833145ba91bbd7c24e68a7e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:51.169806 6733 catalog_manager.cc:5484] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index -1 to 4, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:51.177557 6738 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:51.177744 6738 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:51.180603 6738 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet f259a5eac4354bbc94753fb59cea2a0d
I20240503 01:59:51.181619 7124 tablet_service.cc:1514] Processing DeleteTablet for tablet f259a5eac4354bbc94753fb59cea2a0d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:51 UTC) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:51.181717 9736 tablet_service.cc:1514] Processing DeleteTablet for tablet f259a5eac4354bbc94753fb59cea2a0d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:51 UTC) from {username='slave'} at 127.0.0.1:48152
I20240503 01:59:51.182077 14613 tablet_replica.cc:330] T f259a5eac4354bbc94753fb59cea2a0d P db1a0000aaa346f79c0aa4b6c5105191: stopping tablet replica
I20240503 01:59:51.182077 9342 tablet_service.cc:1514] Processing DeleteTablet for tablet f259a5eac4354bbc94753fb59cea2a0d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:51 UTC) from {username='slave'} at 127.0.0.1:36040
I20240503 01:59:51.182267 14613 raft_consensus.cc:2229] T f259a5eac4354bbc94753fb59cea2a0d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:51.182371 14614 tablet_replica.cc:330] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8: stopping tablet replica
I20240503 01:59:51.182433 14461 tablet_replica.cc:330] T f259a5eac4354bbc94753fb59cea2a0d P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:51.182474 14613 raft_consensus.cc:2258] T f259a5eac4354bbc94753fb59cea2a0d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:51.182626 14461 raft_consensus.cc:2229] T f259a5eac4354bbc94753fb59cea2a0d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:51.182749 14461 raft_consensus.cc:2258] T f259a5eac4354bbc94753fb59cea2a0d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:51.183784 14461 ts_tablet_manager.cc:1875] T f259a5eac4354bbc94753fb59cea2a0d P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:51.182762 14614 raft_consensus.cc:2229] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:51.184105 14614 raft_consensus.cc:2258] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:51.184902 14614 ts_tablet_manager.cc:1875] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:51.184969 14613 ts_tablet_manager.cc:1875] T f259a5eac4354bbc94753fb59cea2a0d P db1a0000aaa346f79c0aa4b6c5105191: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:51.187865 14613 ts_tablet_manager.cc:1888] T f259a5eac4354bbc94753fb59cea2a0d P db1a0000aaa346f79c0aa4b6c5105191: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:51.187981 14461 ts_tablet_manager.cc:1888] T f259a5eac4354bbc94753fb59cea2a0d P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:51.188059 14461 log.cc:1196] T f259a5eac4354bbc94753fb59cea2a0d P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/f259a5eac4354bbc94753fb59cea2a0d
I20240503 01:59:51.188215 14613 log.cc:1196] T f259a5eac4354bbc94753fb59cea2a0d P db1a0000aaa346f79c0aa4b6c5105191: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/f259a5eac4354bbc94753fb59cea2a0d
I20240503 01:59:51.188347 14461 ts_tablet_manager.cc:1909] T f259a5eac4354bbc94753fb59cea2a0d P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting consensus metadata
I20240503 01:59:51.188599 14613 ts_tablet_manager.cc:1909] T f259a5eac4354bbc94753fb59cea2a0d P db1a0000aaa346f79c0aa4b6c5105191: Deleting consensus metadata
I20240503 01:59:51.189203 14614 ts_tablet_manager.cc:1888] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:51.189950 6726 catalog_manager.cc:4830] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): tablet f259a5eac4354bbc94753fb59cea2a0d (table rebalancer_dynamic_table_new_name [id=c56128f83d814e079e5ffe833001a3b7]) successfully deleted
I20240503 01:59:51.191390 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet f259a5eac4354bbc94753fb59cea2a0d (table rebalancer_dynamic_table_new_name [id=c56128f83d814e079e5ffe833001a3b7]) successfully deleted
I20240503 01:59:51.191641 14614 log.cc:1196] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/f259a5eac4354bbc94753fb59cea2a0d
I20240503 01:59:51.192421 14614 ts_tablet_manager.cc:1909] T f259a5eac4354bbc94753fb59cea2a0d P 12e8c683db054df3b75c6285b98a5ad8: Deleting consensus metadata
I20240503 01:59:51.193504 6727 catalog_manager.cc:4830] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): tablet f259a5eac4354bbc94753fb59cea2a0d (table rebalancer_dynamic_table_new_name [id=c56128f83d814e079e5ffe833001a3b7]) successfully deleted
I20240503 01:59:51.197221 9363 consensus_queue.cc:237] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:51.198993 9624 raft_consensus.cc:1261] T 54e3860e5cf54663b4538b9e0f96664e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:51.199141 6882 raft_consensus.cc:1261] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 01:59:51.199553 9298 consensus_peers.cc:485] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 -> Peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): Couldn't send request to peer c688f3df56ac4b6d9b0c30fa4cc8ca2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54e3860e5cf54663b4538b9e0f96664e. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:51.199594 13349 consensus_queue.cc:1035] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:51.200033 14567 consensus_queue.cc:1035] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:51.202430 13349 raft_consensus.cc:2932] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:51.205152 9624 raft_consensus.cc:2932] T 54e3860e5cf54663b4538b9e0f96664e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:51.205457 6882 raft_consensus.cc:2932] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:51.207959 6738 catalog_manager.cc:5484] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:51.283085 6737 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:51.294253 6861 tablet_service.cc:1467] Processing CreateTablet for tablet a19ae31e784745e4922703673febf487 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a241056ca22d482ea7ab125659992aac]), partition=
I20240503 01:59:51.294253 9605 tablet_service.cc:1467] Processing CreateTablet for tablet a19ae31e784745e4922703673febf487 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a241056ca22d482ea7ab125659992aac]), partition=
I20240503 01:59:51.294401 6993 tablet_service.cc:1467] Processing CreateTablet for tablet a19ae31e784745e4922703673febf487 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a241056ca22d482ea7ab125659992aac]), partition=
I20240503 01:59:51.294672 9605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a19ae31e784745e4922703673febf487. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.294672 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a19ae31e784745e4922703673febf487. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.294782 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a19ae31e784745e4922703673febf487. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.297464 14554 tablet_bootstrap.cc:492] T a19ae31e784745e4922703673febf487 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:51.297574 14560 tablet_bootstrap.cc:492] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:51.299291 14554 tablet_bootstrap.cc:654] T a19ae31e784745e4922703673febf487 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:51.300522 14554 tablet_bootstrap.cc:492] T a19ae31e784745e4922703673febf487 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:51.300634 14554 ts_tablet_manager.cc:1367] T a19ae31e784745e4922703673febf487 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240503 01:59:51.301155 14554 raft_consensus.cc:348] T a19ae31e784745e4922703673febf487 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:51.301352 14554 raft_consensus.cc:374] T a19ae31e784745e4922703673febf487 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:51.301434 14554 raft_consensus.cc:729] T a19ae31e784745e4922703673febf487 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.301597 14554 consensus_queue.cc:260] T a19ae31e784745e4922703673febf487 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:51.301869 14359 raft_consensus.cc:482] T 442a974500b8458c951993c5b1547820 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:51.301971 14359 raft_consensus.cc:504] T 442a974500b8458c951993c5b1547820 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:51.302232 14560 tablet_bootstrap.cc:654] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:51.302448 14359 leader_election.cc:288] T 442a974500b8458c951993c5b1547820 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:51.302704 14551 tablet_bootstrap.cc:492] T a19ae31e784745e4922703673febf487 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:51.303359 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442a974500b8458c951993c5b1547820" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:51.303442 14554 ts_tablet_manager.cc:1398] T a19ae31e784745e4922703673febf487 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 01:59:51.303508 9363 raft_consensus.cc:2454] T 442a974500b8458c951993c5b1547820 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:51.303720 14551 tablet_bootstrap.cc:654] T a19ae31e784745e4922703673febf487 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:51.303751 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442a974500b8458c951993c5b1547820" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:51.303860 6882 raft_consensus.cc:2454] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:51.304245 9430 leader_election.cc:302] T 442a974500b8458c951993c5b1547820 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:51.304672 14359 raft_consensus.cc:2781] T 442a974500b8458c951993c5b1547820 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:51.304731 14359 raft_consensus.cc:482] T 442a974500b8458c951993c5b1547820 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:51.304776 14359 raft_consensus.cc:3037] T 442a974500b8458c951993c5b1547820 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:51.306056 14359 raft_consensus.cc:504] T 442a974500b8458c951993c5b1547820 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:51.306493 14359 leader_election.cc:288] T 442a974500b8458c951993c5b1547820 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Requested vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:51.308913 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442a974500b8458c951993c5b1547820" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:51.309113 9363 raft_consensus.cc:3037] T 442a974500b8458c951993c5b1547820 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:51.309634 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442a974500b8458c951993c5b1547820" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:51.309805 6882 raft_consensus.cc:3037] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:51.310566 9363 raft_consensus.cc:2454] T 442a974500b8458c951993c5b1547820 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:51.311141 6882 raft_consensus.cc:2454] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:51.311666 9430 leader_election.cc:302] T 442a974500b8458c951993c5b1547820 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:51.311924 14359 raft_consensus.cc:2781] T 442a974500b8458c951993c5b1547820 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:51.312016 14359 raft_consensus.cc:686] T 442a974500b8458c951993c5b1547820 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Becoming Leader. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Running, Role: LEADER
I20240503 01:59:51.312163 14359 consensus_queue.cc:237] T 442a974500b8458c951993c5b1547820 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:51.317273 14551 tablet_bootstrap.cc:492] T a19ae31e784745e4922703673febf487 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:51.317415 14551 ts_tablet_manager.cc:1367] T a19ae31e784745e4922703673febf487 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 01:59:51.318022 14551 raft_consensus.cc:348] T a19ae31e784745e4922703673febf487 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:51.318149 14551 raft_consensus.cc:374] T a19ae31e784745e4922703673febf487 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:51.318181 14551 raft_consensus.cc:729] T a19ae31e784745e4922703673febf487 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.318320 14551 consensus_queue.cc:260] T a19ae31e784745e4922703673febf487 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:51.319720 14551 ts_tablet_manager.cc:1398] T a19ae31e784745e4922703673febf487 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:51.320592 14560 tablet_bootstrap.cc:492] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:51.320706 14560 ts_tablet_manager.cc:1367] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20240503 01:59:51.321213 14560 raft_consensus.cc:348] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:51.321321 14560 raft_consensus.cc:374] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:51.321349 14560 raft_consensus.cc:729] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.321473 14560 consensus_queue.cc:260] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:51.321792 14560 ts_tablet_manager.cc:1398] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:51.330787 6738 catalog_manager.cc:5484] T 442a974500b8458c951993c5b1547820 P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: term changed from 0 to 1, leader changed from <none> to bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:51.332394 14359 raft_consensus.cc:482] T 6673a223a841415783ed8934d13b20ed P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:51.332684 14359 raft_consensus.cc:504] T 6673a223a841415783ed8934d13b20ed P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:51.333062 14359 leader_election.cc:288] T 6673a223a841415783ed8934d13b20ed P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:51.333532 9624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6673a223a841415783ed8934d13b20ed" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:51.333660 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6673a223a841415783ed8934d13b20ed" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:51.333695 9624 raft_consensus.cc:2454] T 6673a223a841415783ed8934d13b20ed P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:51.333792 7014 raft_consensus.cc:2454] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:51.334219 9427 leader_election.cc:302] T 6673a223a841415783ed8934d13b20ed P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:51.334590 14359 raft_consensus.cc:2781] T 6673a223a841415783ed8934d13b20ed P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:51.334654 14359 raft_consensus.cc:482] T 6673a223a841415783ed8934d13b20ed P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:51.334700 14359 raft_consensus.cc:3037] T 6673a223a841415783ed8934d13b20ed P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:51.335925 14359 raft_consensus.cc:504] T 6673a223a841415783ed8934d13b20ed P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:51.336310 14359 leader_election.cc:288] T 6673a223a841415783ed8934d13b20ed P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Requested vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:51.337246 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6673a223a841415783ed8934d13b20ed" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:51.337375 7012 raft_consensus.cc:3037] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:51.337432 9624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6673a223a841415783ed8934d13b20ed" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:51.337582 9624 raft_consensus.cc:3037] T 6673a223a841415783ed8934d13b20ed P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:51.338358 7012 raft_consensus.cc:2454] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:51.338721 9624 raft_consensus.cc:2454] T 6673a223a841415783ed8934d13b20ed P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:51.338743 9427 leader_election.cc:302] T 6673a223a841415783ed8934d13b20ed P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:51.339609 13671 raft_consensus.cc:2781] T 6673a223a841415783ed8934d13b20ed P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:51.339728 13671 raft_consensus.cc:686] T 6673a223a841415783ed8934d13b20ed P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Becoming Leader. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Running, Role: LEADER
I20240503 01:59:51.339865 13671 consensus_queue.cc:237] T 6673a223a841415783ed8934d13b20ed P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:51.348675 6735 catalog_manager.cc:5484] T 6673a223a841415783ed8934d13b20ed P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: term changed from 0 to 1, leader changed from <none> to bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:51.411623 14558 raft_consensus.cc:482] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:51.411768 14558 raft_consensus.cc:504] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:51.412168 14558 leader_election.cc:288] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:51.412665 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bbbd17c78bc4e329a3a7c7008a34d86" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:51.412688 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bbbd17c78bc4e329a3a7c7008a34d86" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:51.412819 9363 raft_consensus.cc:2454] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 01:59:51.412837 7275 raft_consensus.cc:2454] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 01:59:51.413236 9691 leader_election.cc:302] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:51.413468 14558 raft_consensus.cc:2781] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:51.413533 14558 raft_consensus.cc:482] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:51.413578 14558 raft_consensus.cc:3037] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:51.414746 14558 raft_consensus.cc:504] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:51.415176 14558 leader_election.cc:288] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Requested vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:51.415773 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bbbd17c78bc4e329a3a7c7008a34d86" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:51.416025 7275 raft_consensus.cc:3037] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:51.417235 7275 raft_consensus.cc:2454] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:51.417770 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bbbd17c78bc4e329a3a7c7008a34d86" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:51.417968 9363 raft_consensus.cc:3037] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:51.418190 9691 leader_election.cc:302] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:51.418691 14558 raft_consensus.cc:2781] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:51.418812 14558 raft_consensus.cc:686] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Becoming Leader. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Running, Role: LEADER
I20240503 01:59:51.418972 14558 consensus_queue.cc:237] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:51.419250 9363 raft_consensus.cc:2454] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:51.422972 6734 catalog_manager.cc:5484] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: term changed from 0 to 1, leader changed from <none> to db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:51.452126 9342 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:51.459818 6993 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:51.461017 6861 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:51.461138 7256 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:51.469014 9605 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:51.481081 9736 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:51.491238 7124 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:51.506997 9474 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:51.673910 14243 raft_consensus.cc:482] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:51.674049 14243 raft_consensus.cc:504] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:51.674474 14243 leader_election.cc:288] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:51.674937 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19ae31e784745e4922703673febf487" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:51.674937 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19ae31e784745e4922703673febf487" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:51.675096 9625 raft_consensus.cc:2454] T a19ae31e784745e4922703673febf487 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:51.675096 6881 raft_consensus.cc:2454] T a19ae31e784745e4922703673febf487 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:51.675429 6950 leader_election.cc:302] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:51.675624 14243 raft_consensus.cc:2781] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:51.675689 14243 raft_consensus.cc:482] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:51.675729 14243 raft_consensus.cc:3037] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:51.676908 14243 raft_consensus.cc:504] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:51.677273 14243 leader_election.cc:288] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:51.677642 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19ae31e784745e4922703673febf487" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:51.677680 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a19ae31e784745e4922703673febf487" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:51.677772 9625 raft_consensus.cc:3037] T a19ae31e784745e4922703673febf487 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:51.677803 6881 raft_consensus.cc:3037] T a19ae31e784745e4922703673febf487 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:51.678875 6881 raft_consensus.cc:2454] T a19ae31e784745e4922703673febf487 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:51.678893 9625 raft_consensus.cc:2454] T a19ae31e784745e4922703673febf487 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:51.679270 6949 leader_election.cc:302] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:51.679646 14243 raft_consensus.cc:2781] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:51.679767 14243 raft_consensus.cc:686] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:51.679916 14243 consensus_queue.cc:237] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:51.683184 6734 catalog_manager.cc:5484] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:51.702585 6861 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:51.706229 9342 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:51.708680 14737 ts_tablet_manager.cc:902] T ae9fb6c5833145ba91bbd7c24e68a7e2 P db1a0000aaa346f79c0aa4b6c5105191: Initiating tablet copy from peer 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:51.709300 14737 tablet_copy_client.cc:308] T ae9fb6c5833145ba91bbd7c24e68a7e2 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36271
I20240503 01:59:51.709838 7295 tablet_copy_service.cc:140] P 1d0e738c34af40fa9950c6f086619b3a: Received BeginTabletCopySession request for tablet ae9fb6c5833145ba91bbd7c24e68a7e2 from peer db1a0000aaa346f79c0aa4b6c5105191 ({username='slave'} at 127.0.99.136:38163)
I20240503 01:59:51.709947 7295 tablet_copy_service.cc:161] P 1d0e738c34af40fa9950c6f086619b3a: Beginning new tablet copy session on tablet ae9fb6c5833145ba91bbd7c24e68a7e2 from peer db1a0000aaa346f79c0aa4b6c5105191 at {username='slave'} at 127.0.99.136:38163: session id = db1a0000aaa346f79c0aa4b6c5105191-ae9fb6c5833145ba91bbd7c24e68a7e2
I20240503 01:59:51.710959 7295 tablet_copy_source_session.cc:205] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:51.712082 14737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae9fb6c5833145ba91bbd7c24e68a7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.713701 6993 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:51.715085 14737 tablet_copy_client.cc:783] T ae9fb6c5833145ba91bbd7c24e68a7e2 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:51.715317 14737 tablet_copy_client.cc:647] T ae9fb6c5833145ba91bbd7c24e68a7e2 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:51.717442 9736 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:51.718494 14737 tablet_copy_client.cc:523] T ae9fb6c5833145ba91bbd7c24e68a7e2 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:51.718698 7124 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:51.720223 14737 tablet_bootstrap.cc:492] T ae9fb6c5833145ba91bbd7c24e68a7e2 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:51.728463 9474 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:51.744998 9605 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:51.759286 7256 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:51.761852 14737 tablet_bootstrap.cc:492] T ae9fb6c5833145ba91bbd7c24e68a7e2 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:51.762275 14737 tablet_bootstrap.cc:492] T ae9fb6c5833145ba91bbd7c24e68a7e2 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap complete.
I20240503 01:59:51.762512 14737 ts_tablet_manager.cc:1367] T ae9fb6c5833145ba91bbd7c24e68a7e2 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.042s user 0.027s sys 0.004s
I20240503 01:59:51.762970 14743 ts_tablet_manager.cc:902] T 54e3860e5cf54663b4538b9e0f96664e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Initiating tablet copy from peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:51.763509 14743 tablet_copy_client.cc:308] T 54e3860e5cf54663b4538b9e0f96664e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:37387
I20240503 01:59:51.763854 14737 raft_consensus.cc:348] T ae9fb6c5833145ba91bbd7c24e68a7e2 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:51.764102 9383 tablet_copy_service.cc:140] P 12e8c683db054df3b75c6285b98a5ad8: Received BeginTabletCopySession request for tablet 54e3860e5cf54663b4538b9e0f96664e from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a ({username='slave'} at 127.0.99.131:49995)
I20240503 01:59:51.764338 9383 tablet_copy_service.cc:161] P 12e8c683db054df3b75c6285b98a5ad8: Beginning new tablet copy session on tablet 54e3860e5cf54663b4538b9e0f96664e from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a at {username='slave'} at 127.0.99.131:49995: session id = c688f3df56ac4b6d9b0c30fa4cc8ca2a-54e3860e5cf54663b4538b9e0f96664e
I20240503 01:59:51.764346 14737 raft_consensus.cc:729] T ae9fb6c5833145ba91bbd7c24e68a7e2 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.764600 14737 consensus_queue.cc:260] T ae9fb6c5833145ba91bbd7c24e68a7e2 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:51.765671 9383 tablet_copy_source_session.cc:205] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:51.766815 14743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e3860e5cf54663b4538b9e0f96664e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.767874 14737 ts_tablet_manager.cc:1398] T ae9fb6c5833145ba91bbd7c24e68a7e2 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:51.769882 14743 tablet_copy_client.cc:783] T 54e3860e5cf54663b4538b9e0f96664e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:51.770066 14743 tablet_copy_client.cc:647] T 54e3860e5cf54663b4538b9e0f96664e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:51.770923 7295 tablet_copy_service.cc:342] P 1d0e738c34af40fa9950c6f086619b3a: Request end of tablet copy session db1a0000aaa346f79c0aa4b6c5105191-ae9fb6c5833145ba91bbd7c24e68a7e2 received from {username='slave'} at 127.0.99.136:38163
I20240503 01:59:51.771021 7295 tablet_copy_service.cc:433] P 1d0e738c34af40fa9950c6f086619b3a: ending tablet copy session db1a0000aaa346f79c0aa4b6c5105191-ae9fb6c5833145ba91bbd7c24e68a7e2 on tablet ae9fb6c5833145ba91bbd7c24e68a7e2 with peer db1a0000aaa346f79c0aa4b6c5105191
I20240503 01:59:51.775238 14359 consensus_queue.cc:1035] T 6673a223a841415783ed8934d13b20ed P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:51.790901 14743 tablet_copy_client.cc:523] T 54e3860e5cf54663b4538b9e0f96664e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:51.792878 14743 tablet_bootstrap.cc:492] T 54e3860e5cf54663b4538b9e0f96664e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:51.797916 14743 tablet_bootstrap.cc:492] T 54e3860e5cf54663b4538b9e0f96664e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:51.798182 14743 tablet_bootstrap.cc:492] T 54e3860e5cf54663b4538b9e0f96664e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap complete.
I20240503 01:59:51.798322 14743 ts_tablet_manager.cc:1367] T 54e3860e5cf54663b4538b9e0f96664e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.005s
I20240503 01:59:51.798812 14743 raft_consensus.cc:348] T 54e3860e5cf54663b4538b9e0f96664e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:51.798946 14743 raft_consensus.cc:729] T 54e3860e5cf54663b4538b9e0f96664e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:51.799104 14743 consensus_queue.cc:260] T 54e3860e5cf54663b4538b9e0f96664e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:51.799393 14743 ts_tablet_manager.cc:1398] T 54e3860e5cf54663b4538b9e0f96664e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:51.800251 9383 tablet_copy_service.cc:342] P 12e8c683db054df3b75c6285b98a5ad8: Request end of tablet copy session c688f3df56ac4b6d9b0c30fa4cc8ca2a-54e3860e5cf54663b4538b9e0f96664e received from {username='slave'} at 127.0.99.131:49995
I20240503 01:59:51.800377 9383 tablet_copy_service.cc:433] P 12e8c683db054df3b75c6285b98a5ad8: ending tablet copy session c688f3df56ac4b6d9b0c30fa4cc8ca2a-54e3860e5cf54663b4538b9e0f96664e on tablet 54e3860e5cf54663b4538b9e0f96664e with peer c688f3df56ac4b6d9b0c30fa4cc8ca2a
I20240503 01:59:51.809291 13671 consensus_queue.cc:1035] T 6673a223a841415783ed8934d13b20ed P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:51.831938 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:51.837239 6881 raft_consensus.cc:1261] T a19ae31e784745e4922703673febf487 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:51.837239 9625 raft_consensus.cc:1261] T a19ae31e784745e4922703673febf487 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:51.837787 14004 consensus_queue.cc:1035] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:51.838361 14004 consensus_queue.cc:1035] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:51.841383 6738 catalog_manager.cc:5576] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087) does not have the latest schema for tablet a19ae31e784745e4922703673febf487 (table rebalancer_dynamic_table [id=a241056ca22d482ea7ab125659992aac]). Expected version 1 got 0
I20240503 01:59:51.841794 6734 catalog_manager.cc:5576] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943) does not have the latest schema for tablet a19ae31e784745e4922703673febf487 (table rebalancer_dynamic_table [id=a241056ca22d482ea7ab125659992aac]). Expected version 1 got 0
I20240503 01:59:51.844116 14773 tablet.cc:1722] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:51.844596 14773 tablet.cc:1620] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef: MemRowSet was empty: no flush needed.
I20240503 01:59:51.847113 6734 catalog_manager.cc:5576] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087) does not have the latest schema for tablet a19ae31e784745e4922703673febf487 (table rebalancer_dynamic_table [id=a241056ca22d482ea7ab125659992aac]). Expected version 1 got 0
I20240503 01:59:51.848169 6738 catalog_manager.cc:5576] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943) does not have the latest schema for tablet a19ae31e784745e4922703673febf487 (table rebalancer_dynamic_table [id=a241056ca22d482ea7ab125659992aac]). Expected version 1 got 0
I20240503 01:59:51.850168 14776 tablet.cc:1722] T a19ae31e784745e4922703673febf487 P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:51.850634 14776 tablet.cc:1620] T a19ae31e784745e4922703673febf487 P 5dcc33cc621a4cbbae155f3230fb7c46: MemRowSet was empty: no flush needed.
I20240503 01:59:51.851316 6881 consensus_queue.cc:237] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:51.852962 7012 raft_consensus.cc:1261] T 4e686819aed5407bacc13b3e3a0d4566 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:51.852953 14775 tablet.cc:1722] T a19ae31e784745e4922703673febf487 P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:51.853389 14778 consensus_queue.cc:1035] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:51.853875 14775 tablet.cc:1620] T a19ae31e784745e4922703673febf487 P b7062fbbc2b54fff9e99ec4ea5562c76: MemRowSet was empty: no flush needed.
I20240503 01:59:51.854673 9755 raft_consensus.cc:1261] T 4e686819aed5407bacc13b3e3a0d4566 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:51.854907 6738 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=a241056ca22d482ea7ab125659992aac] alter complete (version 1)
I20240503 01:59:51.855257 14778 consensus_queue.cc:1035] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:51.857120 14473 raft_consensus.cc:2932] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:51.859202 7012 raft_consensus.cc:2932] T 4e686819aed5407bacc13b3e3a0d4566 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:51.860244 9755 raft_consensus.cc:2932] T 4e686819aed5407bacc13b3e3a0d4566 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:51.863313 6734 catalog_manager.cc:5484] T 4e686819aed5407bacc13b3e3a0d4566 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:51.864868 13637 consensus_queue.cc:1035] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:51.868311 9474 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e686819aed5407bacc13b3e3a0d4566 with delete_type TABLET_DATA_TOMBSTONED (TS bb5364d730cb4489a0bfc6c270b7450a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54678
I20240503 01:59:51.871451 14789 tablet_replica.cc:330] T 4e686819aed5407bacc13b3e3a0d4566 P bb5364d730cb4489a0bfc6c270b7450a: stopping tablet replica
I20240503 01:59:51.871376 14558 consensus_queue.cc:1035] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:51.871649 14789 raft_consensus.cc:2229] T 4e686819aed5407bacc13b3e3a0d4566 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:51.871807 14789 raft_consensus.cc:2258] T 4e686819aed5407bacc13b3e3a0d4566 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:51.874652 14789 ts_tablet_manager.cc:1875] T 4e686819aed5407bacc13b3e3a0d4566 P bb5364d730cb4489a0bfc6c270b7450a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:51.877615 14789 ts_tablet_manager.cc:1888] T 4e686819aed5407bacc13b3e3a0d4566 P bb5364d730cb4489a0bfc6c270b7450a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 01:59:51.877700 14789 log.cc:1196] T 4e686819aed5407bacc13b3e3a0d4566 P bb5364d730cb4489a0bfc6c270b7450a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/4e686819aed5407bacc13b3e3a0d4566
I20240503 01:59:51.878275 6725 catalog_manager.cc:4830] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): tablet 4e686819aed5407bacc13b3e3a0d4566 (table rebalancer_extra_table_6 [id=73d68a2fa3594bc4b64c9e73724c9ae0]) successfully deleted
I20240503 01:59:51.884457 13671 consensus_queue.cc:1035] T 442a974500b8458c951993c5b1547820 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:51.888934 14359 consensus_queue.cc:1035] T 442a974500b8458c951993c5b1547820 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:51.959010 6736 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:51.960417 6739 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:51.971547 14773 tablet.cc:1722] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:51.971961 14773 tablet.cc:1620] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef: MemRowSet was empty: no flush needed.
I20240503 01:59:51.973670 14776 tablet.cc:1722] T a19ae31e784745e4922703673febf487 P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:51.974031 14776 tablet.cc:1620] T a19ae31e784745e4922703673febf487 P 5dcc33cc621a4cbbae155f3230fb7c46: MemRowSet was empty: no flush needed.
I20240503 01:59:51.977732 14775 tablet.cc:1722] T a19ae31e784745e4922703673febf487 P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:51.978154 14775 tablet.cc:1620] T a19ae31e784745e4922703673febf487 P b7062fbbc2b54fff9e99ec4ea5562c76: MemRowSet was empty: no flush needed.
I20240503 01:59:51.984690 9474 tablet_service.cc:1467] Processing CreateTablet for tablet 944657ceab07465092a950a8c793ceff (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=2fc3f2f8c5d549c8b1f8c504355aede7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.985044 9474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 944657ceab07465092a950a8c793ceff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.986435 9605 tablet_service.cc:1467] Processing CreateTablet for tablet 6364f3919c42498ca47ba114d525279b (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=2fc3f2f8c5d549c8b1f8c504355aede7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.986692 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 6364f3919c42498ca47ba114d525279b (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=2fc3f2f8c5d549c8b1f8c504355aede7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.986971 9605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6364f3919c42498ca47ba114d525279b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.986981 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6364f3919c42498ca47ba114d525279b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.986946 9604 tablet_service.cc:1467] Processing CreateTablet for tablet 944657ceab07465092a950a8c793ceff (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=2fc3f2f8c5d549c8b1f8c504355aede7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.987224 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 944657ceab07465092a950a8c793ceff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.986708 9341 tablet_service.cc:1467] Processing CreateTablet for tablet 622197c9b8664d94bbae183f6f2fd381 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=2fc3f2f8c5d549c8b1f8c504355aede7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.989712 9341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 622197c9b8664d94bbae183f6f2fd381. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.989696 9606 tablet_service.cc:1467] Processing CreateTablet for tablet 622197c9b8664d94bbae183f6f2fd381 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=2fc3f2f8c5d549c8b1f8c504355aede7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.989923 9606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 622197c9b8664d94bbae183f6f2fd381. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.992647 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 6364f3919c42498ca47ba114d525279b (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=2fc3f2f8c5d549c8b1f8c504355aede7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.992647 6863 tablet_service.cc:1467] Processing CreateTablet for tablet 944657ceab07465092a950a8c793ceff (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=2fc3f2f8c5d549c8b1f8c504355aede7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.993044 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6364f3919c42498ca47ba114d525279b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.993490 6993 tablet_service.cc:1467] Processing CreateTablet for tablet 622197c9b8664d94bbae183f6f2fd381 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=2fc3f2f8c5d549c8b1f8c504355aede7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:51.993757 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 622197c9b8664d94bbae183f6f2fd381. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.994446 6738 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=a241056ca22d482ea7ab125659992aac] alter complete (version 2)
I20240503 01:59:51.995676 14836 tablet_bootstrap.cc:492] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:51.996263 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 944657ceab07465092a950a8c793ceff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:51.997098 14836 tablet_bootstrap.cc:654] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:52.000716 14834 tablet_bootstrap.cc:492] T 6364f3919c42498ca47ba114d525279b P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:52.001048 14838 tablet_bootstrap.cc:492] T 944657ceab07465092a950a8c793ceff P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:52.001361 14835 tablet_bootstrap.cc:492] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:52.002615 14837 tablet_bootstrap.cc:492] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:52.006110 14837 tablet_bootstrap.cc:654] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:52.006314 14838 tablet_bootstrap.cc:654] T 944657ceab07465092a950a8c793ceff P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:52.006999 14835 tablet_bootstrap.cc:654] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:52.006786 14834 tablet_bootstrap.cc:654] T 6364f3919c42498ca47ba114d525279b P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:52.011094 14837 tablet_bootstrap.cc:492] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:52.011219 14837 ts_tablet_manager.cc:1367] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20240503 01:59:52.011283 14835 tablet_bootstrap.cc:492] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:52.011407 14835 ts_tablet_manager.cc:1367] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20240503 01:59:52.012302 14837 raft_consensus.cc:348] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:52.012434 14837 raft_consensus.cc:374] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:52.012466 14837 raft_consensus.cc:729] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:52.012418 14835 raft_consensus.cc:348] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:52.012517 14835 raft_consensus.cc:374] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:52.012539 14835 raft_consensus.cc:729] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:52.012684 14835 consensus_queue.cc:260] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:52.012697 14837 consensus_queue.cc:260] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:52.013044 14837 ts_tablet_manager.cc:1398] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:52.015005 14835 ts_tablet_manager.cc:1398] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 01:59:52.015300 14835 tablet_bootstrap.cc:492] T 6364f3919c42498ca47ba114d525279b P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:52.017032 14834 tablet_bootstrap.cc:492] T 6364f3919c42498ca47ba114d525279b P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:52.017206 14834 ts_tablet_manager.cc:1367] T 6364f3919c42498ca47ba114d525279b P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 01:59:52.017834 14834 raft_consensus.cc:348] T 6364f3919c42498ca47ba114d525279b P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:52.018097 14834 raft_consensus.cc:374] T 6364f3919c42498ca47ba114d525279b P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:52.018153 14834 raft_consensus.cc:729] T 6364f3919c42498ca47ba114d525279b P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:52.018332 14834 consensus_queue.cc:260] T 6364f3919c42498ca47ba114d525279b P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:52.018726 14834 ts_tablet_manager.cc:1398] T 6364f3919c42498ca47ba114d525279b P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:52.019584 14834 tablet_bootstrap.cc:492] T 622197c9b8664d94bbae183f6f2fd381 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:52.020622 14834 tablet_bootstrap.cc:654] T 622197c9b8664d94bbae183f6f2fd381 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:52.021004 14835 tablet_bootstrap.cc:654] T 6364f3919c42498ca47ba114d525279b P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:52.023644 14838 tablet_bootstrap.cc:492] T 944657ceab07465092a950a8c793ceff P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:52.023842 14838 ts_tablet_manager.cc:1367] T 944657ceab07465092a950a8c793ceff P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20240503 01:59:52.024674 14838 raft_consensus.cc:348] T 944657ceab07465092a950a8c793ceff P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:52.024891 14838 raft_consensus.cc:374] T 944657ceab07465092a950a8c793ceff P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:52.024974 14838 raft_consensus.cc:729] T 944657ceab07465092a950a8c793ceff P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:52.026146 14838 consensus_queue.cc:260] T 944657ceab07465092a950a8c793ceff P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:52.027424 14838 ts_tablet_manager.cc:1398] T 944657ceab07465092a950a8c793ceff P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:52.028385 14836 tablet_bootstrap.cc:492] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:52.028498 14836 ts_tablet_manager.cc:1367] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.000s
I20240503 01:59:52.029072 14836 raft_consensus.cc:348] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:52.029204 14836 raft_consensus.cc:374] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:52.029233 14836 raft_consensus.cc:729] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:52.029358 14836 consensus_queue.cc:260] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:52.029646 14836 ts_tablet_manager.cc:1398] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:52.029829 14836 tablet_bootstrap.cc:492] T 944657ceab07465092a950a8c793ceff P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:52.032001 14835 tablet_bootstrap.cc:492] T 6364f3919c42498ca47ba114d525279b P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:52.032193 14835 ts_tablet_manager.cc:1367] T 6364f3919c42498ca47ba114d525279b P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.002s
I20240503 01:59:52.032795 14835 raft_consensus.cc:348] T 6364f3919c42498ca47ba114d525279b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:52.033007 14835 raft_consensus.cc:374] T 6364f3919c42498ca47ba114d525279b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:52.033085 14835 raft_consensus.cc:729] T 6364f3919c42498ca47ba114d525279b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:52.033263 14835 consensus_queue.cc:260] T 6364f3919c42498ca47ba114d525279b P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:52.033458 14836 tablet_bootstrap.cc:654] T 944657ceab07465092a950a8c793ceff P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:52.034453 14835 ts_tablet_manager.cc:1398] T 6364f3919c42498ca47ba114d525279b P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:52.034807 14835 tablet_bootstrap.cc:492] T 622197c9b8664d94bbae183f6f2fd381 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:52.036286 14834 tablet_bootstrap.cc:492] T 622197c9b8664d94bbae183f6f2fd381 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:52.036459 14834 ts_tablet_manager.cc:1367] T 622197c9b8664d94bbae183f6f2fd381 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.002s
I20240503 01:59:52.037343 14834 raft_consensus.cc:348] T 622197c9b8664d94bbae183f6f2fd381 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:52.037573 14834 raft_consensus.cc:374] T 622197c9b8664d94bbae183f6f2fd381 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:52.037670 14834 raft_consensus.cc:729] T 622197c9b8664d94bbae183f6f2fd381 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:52.037931 14835 tablet_bootstrap.cc:654] T 622197c9b8664d94bbae183f6f2fd381 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:52.037987 14834 consensus_queue.cc:260] T 622197c9b8664d94bbae183f6f2fd381 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:52.039168 14834 ts_tablet_manager.cc:1398] T 622197c9b8664d94bbae183f6f2fd381 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:52.040190 14836 tablet_bootstrap.cc:492] T 944657ceab07465092a950a8c793ceff P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:52.040303 14836 ts_tablet_manager.cc:1367] T 944657ceab07465092a950a8c793ceff P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20240503 01:59:52.040858 14836 raft_consensus.cc:348] T 944657ceab07465092a950a8c793ceff P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:52.040984 14836 raft_consensus.cc:374] T 944657ceab07465092a950a8c793ceff P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:52.041080 14836 raft_consensus.cc:729] T 944657ceab07465092a950a8c793ceff P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:52.041229 14836 consensus_queue.cc:260] T 944657ceab07465092a950a8c793ceff P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:52.041561 14836 ts_tablet_manager.cc:1398] T 944657ceab07465092a950a8c793ceff P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:52.042989 9341 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:52.051635 7256 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:52.058633 6863 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:52.071331 6993 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:52.072336 9474 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:52.074587 9736 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:52.075282 14835 tablet_bootstrap.cc:492] T 622197c9b8664d94bbae183f6f2fd381 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:52.076418 14835 ts_tablet_manager.cc:1367] T 622197c9b8664d94bbae183f6f2fd381 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.042s user 0.000s sys 0.002s
I20240503 01:59:52.077112 14835 raft_consensus.cc:348] T 622197c9b8664d94bbae183f6f2fd381 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:52.078441 14835 raft_consensus.cc:374] T 622197c9b8664d94bbae183f6f2fd381 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:52.078584 14835 raft_consensus.cc:729] T 622197c9b8664d94bbae183f6f2fd381 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:52.081745 7124 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:52.100843 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:52.109139 14773 tablet.cc:1722] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:52.113741 14776 tablet.cc:1722] T a19ae31e784745e4922703673febf487 P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:52.120837 6734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=a241056ca22d482ea7ab125659992aac] alter complete (version 3)
I20240503 01:59:52.077451 9606 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 01:59:52.109809 14775 tablet.cc:1722] T a19ae31e784745e4922703673febf487 P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:52.128171 14835 consensus_queue.cc:260] T 622197c9b8664d94bbae183f6f2fd381 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:52.133337 14835 ts_tablet_manager.cc:1398] T 622197c9b8664d94bbae183f6f2fd381 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.057s user 0.000s sys 0.001s
I20240503 01:59:52.203907 9755 consensus_queue.cc:237] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:52.205550 7143 raft_consensus.cc:1261] T fad1113f16ba45009ae409f57eac4550 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:52.205612 9362 raft_consensus.cc:1261] T fad1113f16ba45009ae409f57eac4550 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:52.206142 14558 consensus_queue.cc:1035] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:52.206575 13637 consensus_queue.cc:1035] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:52.208885 13637 raft_consensus.cc:2932] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:52.209432 9362 raft_consensus.cc:2932] T fad1113f16ba45009ae409f57eac4550 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:52.210695 7143 raft_consensus.cc:2932] T fad1113f16ba45009ae409f57eac4550 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:52.212666 6736 catalog_manager.cc:5484] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:52.217186 9606 tablet_service.cc:1514] Processing DeleteTablet for tablet fad1113f16ba45009ae409f57eac4550 with delete_type TABLET_DATA_TOMBSTONED (TS b7062fbbc2b54fff9e99ec4ea5562c76 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:58562
I20240503 01:59:52.217732 14881 tablet_replica.cc:330] T fad1113f16ba45009ae409f57eac4550 P b7062fbbc2b54fff9e99ec4ea5562c76: stopping tablet replica
I20240503 01:59:52.218158 14881 raft_consensus.cc:2229] T fad1113f16ba45009ae409f57eac4550 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:52.218483 14881 raft_consensus.cc:2258] T fad1113f16ba45009ae409f57eac4550 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:52.219293 14881 ts_tablet_manager.cc:1875] T fad1113f16ba45009ae409f57eac4550 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:52.222821 14881 ts_tablet_manager.cc:1888] T fad1113f16ba45009ae409f57eac4550 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 01:59:52.222997 14881 log.cc:1196] T fad1113f16ba45009ae409f57eac4550 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/fad1113f16ba45009ae409f57eac4550
I20240503 01:59:52.223766 6727 catalog_manager.cc:4830] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): tablet fad1113f16ba45009ae409f57eac4550 (table rebalancer_extra_table_12 [id=d46b94d9de0c43ad876391aee881e04d]) successfully deleted
I20240503 01:59:52.224051 6734 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:52.224192 6734 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:52.226454 6734 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet a19ae31e784745e4922703673febf487
I20240503 01:59:52.227137 6993 tablet_service.cc:1514] Processing DeleteTablet for tablet a19ae31e784745e4922703673febf487 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:52 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:52.227511 9606 tablet_service.cc:1514] Processing DeleteTablet for tablet a19ae31e784745e4922703673febf487 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:52 UTC) from {username='slave'} at 127.0.0.1:58562
I20240503 01:59:52.227695 14881 tablet_replica.cc:330] T a19ae31e784745e4922703673febf487 P b7062fbbc2b54fff9e99ec4ea5562c76: stopping tablet replica
I20240503 01:59:52.227706 6863 tablet_service.cc:1514] Processing DeleteTablet for tablet a19ae31e784745e4922703673febf487 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:52 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:52.227782 14889 tablet_replica.cc:330] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:52.227864 14881 raft_consensus.cc:2229] T a19ae31e784745e4922703673febf487 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:52.228009 14881 raft_consensus.cc:2258] T a19ae31e784745e4922703673febf487 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:52.228369 14889 raft_consensus.cc:2229] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:52.228612 14889 raft_consensus.cc:2258] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:52.228688 14881 ts_tablet_manager.cc:1875] T a19ae31e784745e4922703673febf487 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:52.229049 14890 tablet_replica.cc:330] T a19ae31e784745e4922703673febf487 P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:52.229184 14890 raft_consensus.cc:2229] T a19ae31e784745e4922703673febf487 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:52.229288 14890 raft_consensus.cc:2258] T a19ae31e784745e4922703673febf487 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:52.229741 14889 ts_tablet_manager.cc:1875] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:52.229954 14890 ts_tablet_manager.cc:1875] T a19ae31e784745e4922703673febf487 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:52.231374 14881 ts_tablet_manager.cc:1888] T a19ae31e784745e4922703673febf487 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:52.231456 14881 log.cc:1196] T a19ae31e784745e4922703673febf487 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/a19ae31e784745e4922703673febf487
I20240503 01:59:52.231758 14881 ts_tablet_manager.cc:1909] T a19ae31e784745e4922703673febf487 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting consensus metadata
I20240503 01:59:52.232969 6727 catalog_manager.cc:4830] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): tablet a19ae31e784745e4922703673febf487 (table rebalancer_dynamic_table_new_name [id=a241056ca22d482ea7ab125659992aac]) successfully deleted
I20240503 01:59:52.233047 14890 ts_tablet_manager.cc:1888] T a19ae31e784745e4922703673febf487 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:52.233122 14890 log.cc:1196] T a19ae31e784745e4922703673febf487 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/a19ae31e784745e4922703673febf487
I20240503 01:59:52.233482 14890 ts_tablet_manager.cc:1909] T a19ae31e784745e4922703673febf487 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:52.234081 7143 raft_consensus.cc:1203] T 54e3860e5cf54663b4538b9e0f96664e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 01:59:52.234506 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet a19ae31e784745e4922703673febf487 (table rebalancer_dynamic_table_new_name [id=a241056ca22d482ea7ab125659992aac]) successfully deleted
I20240503 01:59:52.236119 14889 ts_tablet_manager.cc:1888] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:52.236205 14889 log.cc:1196] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/a19ae31e784745e4922703673febf487
I20240503 01:59:52.236518 14889 ts_tablet_manager.cc:1909] T a19ae31e784745e4922703673febf487 P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:52.237293 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet a19ae31e784745e4922703673febf487 (table rebalancer_dynamic_table_new_name [id=a241056ca22d482ea7ab125659992aac]) successfully deleted
I20240503 01:59:52.282951 14833 raft_consensus.cc:482] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:52.283105 14833 raft_consensus.cc:504] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:52.283696 14833 leader_election.cc:288] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:52.284229 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "622197c9b8664d94bbae183f6f2fd381" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:52.284257 9362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "622197c9b8664d94bbae183f6f2fd381" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:52.284381 9625 raft_consensus.cc:2454] T 622197c9b8664d94bbae183f6f2fd381 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:52.284399 9362 raft_consensus.cc:2454] T 622197c9b8664d94bbae183f6f2fd381 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:52.284744 6949 leader_election.cc:302] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:52.285431 14833 raft_consensus.cc:2781] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:52.285494 14833 raft_consensus.cc:482] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:52.285543 14833 raft_consensus.cc:3037] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:52.286948 14833 raft_consensus.cc:504] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:52.288796 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "622197c9b8664d94bbae183f6f2fd381" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:52.288945 9625 raft_consensus.cc:3037] T 622197c9b8664d94bbae183f6f2fd381 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:52.289988 9625 raft_consensus.cc:2454] T 622197c9b8664d94bbae183f6f2fd381 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:52.290426 6949 leader_election.cc:302] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:52.290668 14004 raft_consensus.cc:2781] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:52.290766 14004 raft_consensus.cc:686] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:52.290910 14004 consensus_queue.cc:237] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:52.293354 6738 catalog_manager.cc:5484] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:52.294047 14833 leader_election.cc:288] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:52.294486 9362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "622197c9b8664d94bbae183f6f2fd381" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:52.294653 9362 raft_consensus.cc:3037] T 622197c9b8664d94bbae183f6f2fd381 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:52.295818 9362 raft_consensus.cc:2454] T 622197c9b8664d94bbae183f6f2fd381 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:52.306418 9755 raft_consensus.cc:1203] T ae9fb6c5833145ba91bbd7c24e68a7e2 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:52.328276 14781 raft_consensus.cc:482] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:52.328418 14781 raft_consensus.cc:504] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:52.328877 14781 leader_election.cc:288] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:52.329237 9362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364f3919c42498ca47ba114d525279b" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:52.329309 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364f3919c42498ca47ba114d525279b" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:52.329387 9362 raft_consensus.cc:2454] T 6364f3919c42498ca47ba114d525279b P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:52.329469 9625 raft_consensus.cc:2454] T 6364f3919c42498ca47ba114d525279b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:52.329875 6818 leader_election.cc:302] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:52.330231 14781 raft_consensus.cc:2781] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:52.330289 14781 raft_consensus.cc:482] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:52.330332 14781 raft_consensus.cc:3037] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:52.330186 6740 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:52.331502 14781 raft_consensus.cc:504] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:52.331828 14781 leader_election.cc:288] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Requested vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:52.332536 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364f3919c42498ca47ba114d525279b" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:52.332674 9363 raft_consensus.cc:3037] T 6364f3919c42498ca47ba114d525279b P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:52.333194 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364f3919c42498ca47ba114d525279b" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:52.333324 9625 raft_consensus.cc:3037] T 6364f3919c42498ca47ba114d525279b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:52.334177 9363 raft_consensus.cc:2454] T 6364f3919c42498ca47ba114d525279b P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:52.334331 9625 raft_consensus.cc:2454] T 6364f3919c42498ca47ba114d525279b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:52.334741 6818 leader_election.cc:302] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:52.335058 14781 raft_consensus.cc:2781] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:52.335163 14781 raft_consensus.cc:686] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 01:59:52.335314 14781 consensus_queue.cc:237] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:52.339707 7256 tablet_service.cc:1467] Processing CreateTablet for tablet 4741cfb981f6409a98c1ff17992d9416 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2dfe5bbf79244eaabfc8f3bc6082c226]), partition=
I20240503 01:59:52.339080 6740 catalog_manager.cc:5484] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:52.339818 9606 tablet_service.cc:1467] Processing CreateTablet for tablet 4741cfb981f6409a98c1ff17992d9416 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2dfe5bbf79244eaabfc8f3bc6082c226]), partition=
I20240503 01:59:52.339989 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4741cfb981f6409a98c1ff17992d9416. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:52.340054 9606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4741cfb981f6409a98c1ff17992d9416. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:52.342676 14835 tablet_bootstrap.cc:492] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:52.343531 14835 tablet_bootstrap.cc:654] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:52.342473 9736 tablet_service.cc:1467] Processing CreateTablet for tablet 4741cfb981f6409a98c1ff17992d9416 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2dfe5bbf79244eaabfc8f3bc6082c226]), partition=
I20240503 01:59:52.345185 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4741cfb981f6409a98c1ff17992d9416. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:52.349478 14835 tablet_bootstrap.cc:492] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:52.349581 14835 ts_tablet_manager.cc:1367] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240503 01:59:52.350157 14835 raft_consensus.cc:348] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:52.350265 14835 raft_consensus.cc:374] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:52.350293 14835 raft_consensus.cc:729] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:52.350417 14835 consensus_queue.cc:260] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:52.350749 14835 ts_tablet_manager.cc:1398] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:52.354893 14942 tablet_bootstrap.cc:492] T 4741cfb981f6409a98c1ff17992d9416 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:52.356102 14942 tablet_bootstrap.cc:654] T 4741cfb981f6409a98c1ff17992d9416 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:52.358780 14940 tablet_bootstrap.cc:492] T 4741cfb981f6409a98c1ff17992d9416 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:52.360266 14940 tablet_bootstrap.cc:654] T 4741cfb981f6409a98c1ff17992d9416 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:52.361788 14942 tablet_bootstrap.cc:492] T 4741cfb981f6409a98c1ff17992d9416 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:52.361907 14942 ts_tablet_manager.cc:1367] T 4741cfb981f6409a98c1ff17992d9416 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:59:52.362445 14942 raft_consensus.cc:348] T 4741cfb981f6409a98c1ff17992d9416 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:52.362587 14942 raft_consensus.cc:374] T 4741cfb981f6409a98c1ff17992d9416 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:52.362624 14942 raft_consensus.cc:729] T 4741cfb981f6409a98c1ff17992d9416 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:52.362753 14942 consensus_queue.cc:260] T 4741cfb981f6409a98c1ff17992d9416 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:52.363045 14942 ts_tablet_manager.cc:1398] T 4741cfb981f6409a98c1ff17992d9416 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:52.364578 14940 tablet_bootstrap.cc:492] T 4741cfb981f6409a98c1ff17992d9416 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:52.364758 14940 ts_tablet_manager.cc:1367] T 4741cfb981f6409a98c1ff17992d9416 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:52.365301 14940 raft_consensus.cc:348] T 4741cfb981f6409a98c1ff17992d9416 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:52.365515 14940 raft_consensus.cc:374] T 4741cfb981f6409a98c1ff17992d9416 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:52.365602 14940 raft_consensus.cc:729] T 4741cfb981f6409a98c1ff17992d9416 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:52.366014 14940 consensus_queue.cc:260] T 4741cfb981f6409a98c1ff17992d9416 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:52.366379 14940 ts_tablet_manager.cc:1398] T 4741cfb981f6409a98c1ff17992d9416 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:52.377040 14945 raft_consensus.cc:482] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:52.377254 14945 raft_consensus.cc:504] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:52.378638 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "944657ceab07465092a950a8c793ceff" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:52.378791 14945 leader_election.cc:288] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:52.378829 9493 raft_consensus.cc:2454] T 944657ceab07465092a950a8c793ceff P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 01:59:52.379141 9559 leader_election.cc:302] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:52.379315 14945 raft_consensus.cc:2781] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:52.379377 14945 raft_consensus.cc:482] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:52.379451 14945 raft_consensus.cc:3037] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:52.379417 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "944657ceab07465092a950a8c793ceff" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:52.379549 6882 raft_consensus.cc:2454] T 944657ceab07465092a950a8c793ceff P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 01:59:52.380563 14945 raft_consensus.cc:504] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:52.381181 14945 leader_election.cc:288] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Requested vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:52.381578 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "944657ceab07465092a950a8c793ceff" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:52.381708 6882 raft_consensus.cc:3037] T 944657ceab07465092a950a8c793ceff P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:52.381721 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "944657ceab07465092a950a8c793ceff" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:52.381934 9493 raft_consensus.cc:3037] T 944657ceab07465092a950a8c793ceff P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:52.382709 6882 raft_consensus.cc:2454] T 944657ceab07465092a950a8c793ceff P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 01:59:52.383093 9493 raft_consensus.cc:2454] T 944657ceab07465092a950a8c793ceff P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 01:59:52.383201 9562 leader_election.cc:302] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:52.383443 14945 raft_consensus.cc:2781] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:52.383688 14945 raft_consensus.cc:686] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 LEADER]: Becoming Leader. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Running, Role: LEADER
I20240503 01:59:52.384224 14945 consensus_queue.cc:237] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:52.387454 6739 catalog_manager.cc:5484] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:52.408288 6863 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:52.413866 9474 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:52.414783 7256 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:52.424990 7124 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:52.425202 9606 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:52.426198 9341 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:52.434440 14945 raft_consensus.cc:482] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:52.434618 14945 raft_consensus.cc:504] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:52.435043 14945 leader_election.cc:288] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:52.439435 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4741cfb981f6409a98c1ff17992d9416" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:52.439466 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4741cfb981f6409a98c1ff17992d9416" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:52.439611 9757 raft_consensus.cc:2454] T 4741cfb981f6409a98c1ff17992d9416 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 01:59:52.439642 7275 raft_consensus.cc:2454] T 4741cfb981f6409a98c1ff17992d9416 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 01:59:52.439918 9560 leader_election.cc:302] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:52.440110 14945 raft_consensus.cc:2781] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:52.440171 14945 raft_consensus.cc:482] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:52.440205 14945 raft_consensus.cc:3037] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:52.441346 14945 raft_consensus.cc:504] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:52.441763 14945 leader_election.cc:288] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:52.442600 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4741cfb981f6409a98c1ff17992d9416" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:52.442739 9757 raft_consensus.cc:3037] T 4741cfb981f6409a98c1ff17992d9416 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:52.442878 9736 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:52.443840 9757 raft_consensus.cc:2454] T 4741cfb981f6409a98c1ff17992d9416 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 01:59:52.445822 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4741cfb981f6409a98c1ff17992d9416" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:52.445971 7275 raft_consensus.cc:3037] T 4741cfb981f6409a98c1ff17992d9416 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:52.447170 7275 raft_consensus.cc:2454] T 4741cfb981f6409a98c1ff17992d9416 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 01:59:52.448340 9560 leader_election.cc:302] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:52.449283 14945 raft_consensus.cc:2781] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:52.449373 14945 raft_consensus.cc:686] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 LEADER]: Becoming Leader. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Running, Role: LEADER
I20240503 01:59:52.449506 14945 consensus_queue.cc:237] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:52.453601 6993 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:52.455809 6737 catalog_manager.cc:5484] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:52.525467 7275 consensus_queue.cc:237] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:52.526906 9755 raft_consensus.cc:1261] T ae9fb6c5833145ba91bbd7c24e68a7e2 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:52.526929 7143 raft_consensus.cc:1261] T ae9fb6c5833145ba91bbd7c24e68a7e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:52.527362 14304 consensus_queue.cc:1035] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:52.527489 14599 consensus_queue.cc:1035] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:52.529289 14304 raft_consensus.cc:2932] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:52.529778 7143 raft_consensus.cc:2932] T ae9fb6c5833145ba91bbd7c24e68a7e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:52.532300 9755 raft_consensus.cc:2932] T ae9fb6c5833145ba91bbd7c24e68a7e2 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:52.533349 6737 catalog_manager.cc:5484] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:52.537102 6863 tablet_service.cc:1514] Processing DeleteTablet for tablet ae9fb6c5833145ba91bbd7c24e68a7e2 with delete_type TABLET_DATA_TOMBSTONED (TS 5dcc33cc621a4cbbae155f3230fb7c46 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:52.537393 14890 tablet_replica.cc:330] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:52.537586 14890 raft_consensus.cc:2229] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:52.537752 14890 raft_consensus.cc:2258] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:52.538415 14890 ts_tablet_manager.cc:1875] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:52.540895 14890 ts_tablet_manager.cc:1888] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 01:59:52.540992 14890 log.cc:1196] T ae9fb6c5833145ba91bbd7c24e68a7e2 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/ae9fb6c5833145ba91bbd7c24e68a7e2
I20240503 01:59:52.542011 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet ae9fb6c5833145ba91bbd7c24e68a7e2 (table rebalance_test_table_1 [id=38e214942900477190e2c3b67ccc5952]) successfully deleted
I20240503 01:59:52.566289 6739 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:52.571446 7275 raft_consensus.cc:1261] T 4741cfb981f6409a98c1ff17992d9416 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer b7062fbbc2b54fff9e99ec4ea5562c76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:52.573169 9755 raft_consensus.cc:1261] T 4741cfb981f6409a98c1ff17992d9416 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer b7062fbbc2b54fff9e99ec4ea5562c76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:52.573207 14945 consensus_queue.cc:1035] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:52.573915 14945 consensus_queue.cc:1035] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:52.576825 6739 catalog_manager.cc:5576] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271) does not have the latest schema for tablet 4741cfb981f6409a98c1ff17992d9416 (table rebalancer_dynamic_table [id=2dfe5bbf79244eaabfc8f3bc6082c226]). Expected version 1 got 0
I20240503 01:59:52.578182 14775 tablet.cc:1722] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:52.578596 14775 tablet.cc:1620] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76: MemRowSet was empty: no flush needed.
I20240503 01:59:52.580142 6738 catalog_manager.cc:5576] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053) does not have the latest schema for tablet 4741cfb981f6409a98c1ff17992d9416 (table rebalancer_dynamic_table [id=2dfe5bbf79244eaabfc8f3bc6082c226]). Expected version 1 got 0
I20240503 01:59:52.580602 14993 tablet.cc:1722] T 4741cfb981f6409a98c1ff17992d9416 P 1d0e738c34af40fa9950c6f086619b3a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:52.581009 14993 tablet.cc:1620] T 4741cfb981f6409a98c1ff17992d9416 P 1d0e738c34af40fa9950c6f086619b3a: MemRowSet was empty: no flush needed.
I20240503 01:59:52.581840 6738 catalog_manager.cc:5576] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271) does not have the latest schema for tablet 4741cfb981f6409a98c1ff17992d9416 (table rebalancer_dynamic_table [id=2dfe5bbf79244eaabfc8f3bc6082c226]). Expected version 1 got 0
I20240503 01:59:52.588454 14998 tablet.cc:1722] T 4741cfb981f6409a98c1ff17992d9416 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:52.588922 6738 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=2dfe5bbf79244eaabfc8f3bc6082c226] alter complete (version 1)
I20240503 01:59:52.588951 14998 tablet.cc:1620] T 4741cfb981f6409a98c1ff17992d9416 P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:52.590135 6739 catalog_manager.cc:5576] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053) does not have the latest schema for tablet 4741cfb981f6409a98c1ff17992d9416 (table rebalancer_dynamic_table [id=2dfe5bbf79244eaabfc8f3bc6082c226]). Expected version 1 got 0
I20240503 01:59:52.672883 6993 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:52.678611 9606 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:52.679490 9341 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:52.686183 9736 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:52.686504 6863 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:52.689410 7256 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:52.691954 7124 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:52.695329 6737 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:52.699704 14833 consensus_queue.cc:1035] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:52.704653 14775 tablet.cc:1722] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:52.705046 14775 tablet.cc:1620] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76: MemRowSet was empty: no flush needed.
I20240503 01:59:52.713929 14993 tablet.cc:1722] T 4741cfb981f6409a98c1ff17992d9416 P 1d0e738c34af40fa9950c6f086619b3a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:52.714313 14993 tablet.cc:1620] T 4741cfb981f6409a98c1ff17992d9416 P 1d0e738c34af40fa9950c6f086619b3a: MemRowSet was empty: no flush needed.
I20240503 01:59:52.714666 14998 tablet.cc:1722] T 4741cfb981f6409a98c1ff17992d9416 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:52.714962 14998 tablet.cc:1620] T 4741cfb981f6409a98c1ff17992d9416 P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:52.718545 6734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=2dfe5bbf79244eaabfc8f3bc6082c226] alter complete (version 2)
I20240503 01:59:52.722512 14833 consensus_queue.cc:1035] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:52.759066 9474 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 01:59:52.815485 14781 consensus_queue.cc:1035] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:52.815622 7145 consensus_queue.cc:237] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:52.817008 9755 raft_consensus.cc:1261] T 84812ece4f27484eb008bfbebdbfee84 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:52.817072 7275 raft_consensus.cc:1261] T 84812ece4f27484eb008bfbebdbfee84 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:52.817412 14333 consensus_queue.cc:1035] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:52.817698 13734 consensus_queue.cc:1035] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:52.819671 14782 consensus_queue.cc:1035] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:52.820539 13734 raft_consensus.cc:2932] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:52.822906 6732 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:52.822917 9755 raft_consensus.cc:2932] T 84812ece4f27484eb008bfbebdbfee84 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:52.823372 7275 raft_consensus.cc:2932] T 84812ece4f27484eb008bfbebdbfee84 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:52.827813 6737 catalog_manager.cc:5484] T 84812ece4f27484eb008bfbebdbfee84 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:52.831338 9341 tablet_service.cc:1514] Processing DeleteTablet for tablet 84812ece4f27484eb008bfbebdbfee84 with delete_type TABLET_DATA_TOMBSTONED (TS 12e8c683db054df3b75c6285b98a5ad8 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:36040
I20240503 01:59:52.833108 15062 tablet_replica.cc:330] T 84812ece4f27484eb008bfbebdbfee84 P 12e8c683db054df3b75c6285b98a5ad8: stopping tablet replica
I20240503 01:59:52.833570 15062 raft_consensus.cc:2229] T 84812ece4f27484eb008bfbebdbfee84 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:52.833735 15062 raft_consensus.cc:2258] T 84812ece4f27484eb008bfbebdbfee84 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:52.834256 15062 ts_tablet_manager.cc:1875] T 84812ece4f27484eb008bfbebdbfee84 P 12e8c683db054df3b75c6285b98a5ad8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:52.836592 14775 tablet.cc:1722] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:52.836885 15062 ts_tablet_manager.cc:1888] T 84812ece4f27484eb008bfbebdbfee84 P 12e8c683db054df3b75c6285b98a5ad8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 01:59:52.837004 15062 log.cc:1196] T 84812ece4f27484eb008bfbebdbfee84 P 12e8c683db054df3b75c6285b98a5ad8: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/84812ece4f27484eb008bfbebdbfee84
I20240503 01:59:52.837692 14998 tablet.cc:1722] T 4741cfb981f6409a98c1ff17992d9416 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:52.839628 6727 catalog_manager.cc:4830] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): tablet 84812ece4f27484eb008bfbebdbfee84 (table rebalancer_extra_table_10 [id=15afeaf3f27142ebaf9ef93cac9e4dd6]) successfully deleted
I20240503 01:59:52.843283 6736 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=2dfe5bbf79244eaabfc8f3bc6082c226] alter complete (version 3)
I20240503 01:59:52.843917 14993 tablet.cc:1722] T 4741cfb981f6409a98c1ff17992d9416 P 1d0e738c34af40fa9950c6f086619b3a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:52.883615 14945 consensus_queue.cc:1035] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:52.888439 14749 consensus_queue.cc:1035] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:52.951512 6740 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:52.951699 6740 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:52.954375 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 4741cfb981f6409a98c1ff17992d9416
I20240503 01:59:52.955224 9606 tablet_service.cc:1514] Processing DeleteTablet for tablet 4741cfb981f6409a98c1ff17992d9416 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:52 UTC) from {username='slave'} at 127.0.0.1:58562
I20240503 01:59:52.955330 7256 tablet_service.cc:1514] Processing DeleteTablet for tablet 4741cfb981f6409a98c1ff17992d9416 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:52 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:52.955579 9736 tablet_service.cc:1514] Processing DeleteTablet for tablet 4741cfb981f6409a98c1ff17992d9416 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:52 UTC) from {username='slave'} at 127.0.0.1:48152
I20240503 01:59:52.955996 15132 tablet_replica.cc:330] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76: stopping tablet replica
I20240503 01:59:52.955996 15133 tablet_replica.cc:330] T 4741cfb981f6409a98c1ff17992d9416 P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:52.956166 15133 raft_consensus.cc:2229] T 4741cfb981f6409a98c1ff17992d9416 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:52.956210 15132 raft_consensus.cc:2229] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:52.956247 15134 tablet_replica.cc:330] T 4741cfb981f6409a98c1ff17992d9416 P db1a0000aaa346f79c0aa4b6c5105191: stopping tablet replica
I20240503 01:59:52.956357 15133 raft_consensus.cc:2258] T 4741cfb981f6409a98c1ff17992d9416 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:52.956456 15134 raft_consensus.cc:2229] T 4741cfb981f6409a98c1ff17992d9416 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:52.956537 15132 raft_consensus.cc:2258] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:52.956601 15134 raft_consensus.cc:2258] T 4741cfb981f6409a98c1ff17992d9416 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:52.957264 15132 ts_tablet_manager.cc:1875] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:52.957473 15134 ts_tablet_manager.cc:1875] T 4741cfb981f6409a98c1ff17992d9416 P db1a0000aaa346f79c0aa4b6c5105191: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:52.958032 15133 ts_tablet_manager.cc:1875] T 4741cfb981f6409a98c1ff17992d9416 P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:52.959916 15132 ts_tablet_manager.cc:1888] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:52.960130 15132 log.cc:1196] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/4741cfb981f6409a98c1ff17992d9416
I20240503 01:59:52.960433 15132 ts_tablet_manager.cc:1909] T 4741cfb981f6409a98c1ff17992d9416 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting consensus metadata
I20240503 01:59:52.961297 6727 catalog_manager.cc:4830] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): tablet 4741cfb981f6409a98c1ff17992d9416 (table rebalancer_dynamic_table_new_name [id=2dfe5bbf79244eaabfc8f3bc6082c226]) successfully deleted
I20240503 01:59:52.961254 6740 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:52.961483 15134 ts_tablet_manager.cc:1888] T 4741cfb981f6409a98c1ff17992d9416 P db1a0000aaa346f79c0aa4b6c5105191: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:52.961596 15134 log.cc:1196] T 4741cfb981f6409a98c1ff17992d9416 P db1a0000aaa346f79c0aa4b6c5105191: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/4741cfb981f6409a98c1ff17992d9416
I20240503 01:59:52.962006 15134 ts_tablet_manager.cc:1909] T 4741cfb981f6409a98c1ff17992d9416 P db1a0000aaa346f79c0aa4b6c5105191: Deleting consensus metadata
I20240503 01:59:52.963182 6726 catalog_manager.cc:4830] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): tablet 4741cfb981f6409a98c1ff17992d9416 (table rebalancer_dynamic_table_new_name [id=2dfe5bbf79244eaabfc8f3bc6082c226]) successfully deleted
I20240503 01:59:52.964010 15133 ts_tablet_manager.cc:1888] T 4741cfb981f6409a98c1ff17992d9416 P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:52.964092 15133 log.cc:1196] T 4741cfb981f6409a98c1ff17992d9416 P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/4741cfb981f6409a98c1ff17992d9416
I20240503 01:59:52.964442 15133 ts_tablet_manager.cc:1909] T 4741cfb981f6409a98c1ff17992d9416 P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:52.967074 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet 4741cfb981f6409a98c1ff17992d9416 (table rebalancer_dynamic_table_new_name [id=2dfe5bbf79244eaabfc8f3bc6082c226]) successfully deleted
I20240503 01:59:52.969338 9474 tablet_service.cc:1467] Processing CreateTablet for tablet 5089809a9f3846f5b30397b1062d4e55 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=28d2e532bc114d04b0b1e87094387600]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:52.969513 9606 tablet_service.cc:1467] Processing CreateTablet for tablet 5089809a9f3846f5b30397b1062d4e55 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=28d2e532bc114d04b0b1e87094387600]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:52.969703 9474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5089809a9f3846f5b30397b1062d4e55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:52.969797 9606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5089809a9f3846f5b30397b1062d4e55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:52.969818 9341 tablet_service.cc:1467] Processing CreateTablet for tablet 5089809a9f3846f5b30397b1062d4e55 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=28d2e532bc114d04b0b1e87094387600]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:52.970080 9341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5089809a9f3846f5b30397b1062d4e55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:52.972391 15140 tablet_bootstrap.cc:492] T 5089809a9f3846f5b30397b1062d4e55 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:52.972482 9473 tablet_service.cc:1467] Processing CreateTablet for tablet c0e7400f30244b4785c3389c63c7f9d6 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=28d2e532bc114d04b0b1e87094387600]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:52.972748 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0e7400f30244b4785c3389c63c7f9d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:52.972811 6993 tablet_service.cc:1467] Processing CreateTablet for tablet d4fa0337228d4f8cb6b6c3aab437619e (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=28d2e532bc114d04b0b1e87094387600]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:52.973094 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4fa0337228d4f8cb6b6c3aab437619e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:52.973546 15140 tablet_bootstrap.cc:654] T 5089809a9f3846f5b30397b1062d4e55 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:52.974171 15142 tablet_bootstrap.cc:492] T 5089809a9f3846f5b30397b1062d4e55 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:52.974853 9472 tablet_service.cc:1467] Processing CreateTablet for tablet d4fa0337228d4f8cb6b6c3aab437619e (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=28d2e532bc114d04b0b1e87094387600]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:52.975090 9472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4fa0337228d4f8cb6b6c3aab437619e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:52.975217 15142 tablet_bootstrap.cc:654] T 5089809a9f3846f5b30397b1062d4e55 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:52.975320 9736 tablet_service.cc:1467] Processing CreateTablet for tablet c0e7400f30244b4785c3389c63c7f9d6 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=28d2e532bc114d04b0b1e87094387600]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:52.975602 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0e7400f30244b4785c3389c63c7f9d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:52.975661 15144 tablet_bootstrap.cc:492] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:52.976785 15142 tablet_bootstrap.cc:492] T 5089809a9f3846f5b30397b1062d4e55 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:52.976882 15142 ts_tablet_manager.cc:1367] T 5089809a9f3846f5b30397b1062d4e55 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:52.977125 15144 tablet_bootstrap.cc:654] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:52.977381 15142 raft_consensus.cc:348] T 5089809a9f3846f5b30397b1062d4e55 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:52.977664 15142 raft_consensus.cc:374] T 5089809a9f3846f5b30397b1062d4e55 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:52.977735 15142 raft_consensus.cc:729] T 5089809a9f3846f5b30397b1062d4e55 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:52.977663 9604 tablet_service.cc:1467] Processing CreateTablet for tablet d4fa0337228d4f8cb6b6c3aab437619e (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=28d2e532bc114d04b0b1e87094387600]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:52.977871 15142 consensus_queue.cc:260] T 5089809a9f3846f5b30397b1062d4e55 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:52.977952 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4fa0337228d4f8cb6b6c3aab437619e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:52.978382 15146 tablet_bootstrap.cc:492] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:52.979283 15146 tablet_bootstrap.cc:654] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:52.979362 15142 ts_tablet_manager.cc:1398] T 5089809a9f3846f5b30397b1062d4e55 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:52.980219 15146 tablet_bootstrap.cc:492] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:52.980253 15140 tablet_bootstrap.cc:492] T 5089809a9f3846f5b30397b1062d4e55 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:52.980309 15146 ts_tablet_manager.cc:1367] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:52.980340 15140 ts_tablet_manager.cc:1367] T 5089809a9f3846f5b30397b1062d4e55 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 01:59:52.980940 15140 raft_consensus.cc:348] T 5089809a9f3846f5b30397b1062d4e55 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:52.981050 15140 raft_consensus.cc:374] T 5089809a9f3846f5b30397b1062d4e55 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:52.981084 15140 raft_consensus.cc:729] T 5089809a9f3846f5b30397b1062d4e55 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:52.981237 15140 consensus_queue.cc:260] T 5089809a9f3846f5b30397b1062d4e55 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:52.981278 15146 raft_consensus.cc:348] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:52.981389 15146 raft_consensus.cc:374] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:52.981420 15146 raft_consensus.cc:729] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:52.981568 15146 consensus_queue.cc:260] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:52.981868 15146 ts_tablet_manager.cc:1398] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:52.982450 15144 tablet_bootstrap.cc:492] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:52.982470 15140 ts_tablet_manager.cc:1398] T 5089809a9f3846f5b30397b1062d4e55 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:52.982578 15144 ts_tablet_manager.cc:1367] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:59:52.982724 15140 tablet_bootstrap.cc:492] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:52.982996 15144 raft_consensus.cc:348] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:52.983083 15144 raft_consensus.cc:374] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:52.983103 15144 raft_consensus.cc:729] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:52.983224 15144 consensus_queue.cc:260] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:52.983511 15144 ts_tablet_manager.cc:1398] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:52.984058 9341 tablet_service.cc:1467] Processing CreateTablet for tablet c0e7400f30244b4785c3389c63c7f9d6 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=28d2e532bc114d04b0b1e87094387600]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:52.984361 9341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0e7400f30244b4785c3389c63c7f9d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:52.985141 15141 tablet_bootstrap.cc:492] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:52.986276 15141 tablet_bootstrap.cc:654] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:52.988457 15142 tablet_bootstrap.cc:492] T c0e7400f30244b4785c3389c63c7f9d6 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:52.990104 15142 tablet_bootstrap.cc:654] T c0e7400f30244b4785c3389c63c7f9d6 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:52.991725 15142 tablet_bootstrap.cc:492] T c0e7400f30244b4785c3389c63c7f9d6 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:52.991910 15142 ts_tablet_manager.cc:1367] T c0e7400f30244b4785c3389c63c7f9d6 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:52.992496 15142 raft_consensus.cc:348] T c0e7400f30244b4785c3389c63c7f9d6 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:52.992723 15142 raft_consensus.cc:374] T c0e7400f30244b4785c3389c63c7f9d6 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:52.992815 15142 raft_consensus.cc:729] T c0e7400f30244b4785c3389c63c7f9d6 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:52.993008 15142 consensus_queue.cc:260] T c0e7400f30244b4785c3389c63c7f9d6 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:52.993393 15142 ts_tablet_manager.cc:1398] T c0e7400f30244b4785c3389c63c7f9d6 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:52.994652 15140 tablet_bootstrap.cc:654] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:52.996111 15141 tablet_bootstrap.cc:492] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:52.996238 15141 ts_tablet_manager.cc:1367] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 01:59:52.996829 15140 tablet_bootstrap.cc:492] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:52.996940 15140 ts_tablet_manager.cc:1367] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20240503 01:59:52.996855 15141 raft_consensus.cc:348] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:52.997435 15140 raft_consensus.cc:348] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:52.997573 15140 raft_consensus.cc:374] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:52.997604 15140 raft_consensus.cc:729] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:52.997725 15140 consensus_queue.cc:260] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:52.998018 15140 ts_tablet_manager.cc:1398] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:52.998924 15141 raft_consensus.cc:374] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:52.999050 15141 raft_consensus.cc:729] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:52.999398 15141 consensus_queue.cc:260] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:52.999938 15141 ts_tablet_manager.cc:1398] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:53.000257 15141 tablet_bootstrap.cc:492] T d4fa0337228d4f8cb6b6c3aab437619e P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:53.001814 15141 tablet_bootstrap.cc:654] T d4fa0337228d4f8cb6b6c3aab437619e P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.003583 15096 raft_consensus.cc:482] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:53.003695 15096 raft_consensus.cc:504] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:53.003917 15141 tablet_bootstrap.cc:492] T d4fa0337228d4f8cb6b6c3aab437619e P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:53.003855 7145 consensus_queue.cc:237] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:53.004017 15141 ts_tablet_manager.cc:1367] T d4fa0337228d4f8cb6b6c3aab437619e P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 01:59:53.004174 15096 leader_election.cc:288] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:53.004570 15141 raft_consensus.cc:348] T d4fa0337228d4f8cb6b6c3aab437619e P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:53.004721 15141 raft_consensus.cc:374] T d4fa0337228d4f8cb6b6c3aab437619e P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.004797 15141 raft_consensus.cc:729] T d4fa0337228d4f8cb6b6c3aab437619e P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.004869 9624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5089809a9f3846f5b30397b1062d4e55" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:53.004940 15141 consensus_queue.cc:260] T d4fa0337228d4f8cb6b6c3aab437619e P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:53.005013 9624 raft_consensus.cc:2454] T 5089809a9f3846f5b30397b1062d4e55 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:53.005277 15141 ts_tablet_manager.cc:1398] T d4fa0337228d4f8cb6b6c3aab437619e P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:53.005343 9429 leader_election.cc:302] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:53.005509 15096 raft_consensus.cc:2781] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:53.005555 15096 raft_consensus.cc:482] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:53.005578 15141 tablet_bootstrap.cc:492] T c0e7400f30244b4785c3389c63c7f9d6 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:53.005793 9362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5089809a9f3846f5b30397b1062d4e55" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:53.005839 7275 raft_consensus.cc:1261] T 822bafcbc6de406997955ff7a1d9b612 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:53.005936 9362 raft_consensus.cc:2454] T 5089809a9f3846f5b30397b1062d4e55 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:53.006119 6882 raft_consensus.cc:1261] T 822bafcbc6de406997955ff7a1d9b612 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:53.006280 14333 consensus_queue.cc:1035] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:53.006744 15141 tablet_bootstrap.cc:654] T c0e7400f30244b4785c3389c63c7f9d6 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
W20240503 01:59:53.007221 7078 consensus_peers.cc:485] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 822bafcbc6de406997955ff7a1d9b612. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:53.007347 14333 consensus_queue.cc:1035] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:53.005587 15096 raft_consensus.cc:3037] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.007961 15141 tablet_bootstrap.cc:492] T c0e7400f30244b4785c3389c63c7f9d6 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:53.008035 15141 ts_tablet_manager.cc:1367] T c0e7400f30244b4785c3389c63c7f9d6 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 01:59:53.008445 15141 raft_consensus.cc:348] T c0e7400f30244b4785c3389c63c7f9d6 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:53.008536 15141 raft_consensus.cc:374] T c0e7400f30244b4785c3389c63c7f9d6 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.008566 15141 raft_consensus.cc:729] T c0e7400f30244b4785c3389c63c7f9d6 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.008713 15141 consensus_queue.cc:260] T c0e7400f30244b4785c3389c63c7f9d6 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:53.008960 15096 raft_consensus.cc:504] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:53.009033 15141 ts_tablet_manager.cc:1398] T c0e7400f30244b4785c3389c63c7f9d6 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:53.009316 15096 leader_election.cc:288] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Requested vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:53.009685 9362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5089809a9f3846f5b30397b1062d4e55" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:53.009737 9624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5089809a9f3846f5b30397b1062d4e55" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:53.009816 9362 raft_consensus.cc:3037] T 5089809a9f3846f5b30397b1062d4e55 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.009861 9624 raft_consensus.cc:3037] T 5089809a9f3846f5b30397b1062d4e55 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.009860 13734 raft_consensus.cc:2932] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:53.010351 7275 raft_consensus.cc:2932] T 822bafcbc6de406997955ff7a1d9b612 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:53.010957 9362 raft_consensus.cc:2454] T 5089809a9f3846f5b30397b1062d4e55 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:53.011375 9429 leader_election.cc:302] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:53.011592 13671 raft_consensus.cc:2781] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:53.011675 13671 raft_consensus.cc:686] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Becoming Leader. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Running, Role: LEADER
I20240503 01:59:53.011775 9624 raft_consensus.cc:2454] T 5089809a9f3846f5b30397b1062d4e55 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:53.011809 13671 consensus_queue.cc:237] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:53.012524 6882 raft_consensus.cc:2932] T 822bafcbc6de406997955ff7a1d9b612 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:53.016032 6734 catalog_manager.cc:5484] T 822bafcbc6de406997955ff7a1d9b612 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:53.023000 6732 catalog_manager.cc:5484] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: term changed from 0 to 1, leader changed from <none> to bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:53.050685 6882 consensus_queue.cc:237] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:53.052489 7145 raft_consensus.cc:1261] T dcc773c6d5794e26a0003c7d4acc5d21 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:53.052935 7275 raft_consensus.cc:1261] T dcc773c6d5794e26a0003c7d4acc5d21 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:53.052922 15182 consensus_queue.cc:1035] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:53.053289 6818 consensus_peers.cc:485] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dcc773c6d5794e26a0003c7d4acc5d21. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:53.053512 15182 consensus_queue.cc:1035] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:53.055898 14781 raft_consensus.cc:2932] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:53.057056 7145 raft_consensus.cc:2932] T dcc773c6d5794e26a0003c7d4acc5d21 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:53.058192 6732 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:53.058164 7275 raft_consensus.cc:2932] T dcc773c6d5794e26a0003c7d4acc5d21 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:53.060384 14749 raft_consensus.cc:482] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:53.060511 14749 raft_consensus.cc:504] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:53.060967 14749 leader_election.cc:288] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:53.061504 6734 catalog_manager.cc:5484] T dcc773c6d5794e26a0003c7d4acc5d21 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:53.062575 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fa0337228d4f8cb6b6c3aab437619e" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:53.062726 9493 raft_consensus.cc:2454] T d4fa0337228d4f8cb6b6c3aab437619e P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 01:59:53.062978 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fa0337228d4f8cb6b6c3aab437619e" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:53.063104 7014 raft_consensus.cc:2454] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 01:59:53.063143 9559 leader_election.cc:302] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:53.063432 14749 raft_consensus.cc:2781] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:53.063498 14749 raft_consensus.cc:482] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:53.063546 14749 raft_consensus.cc:3037] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.064469 14749 raft_consensus.cc:504] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:53.064899 14749 leader_election.cc:288] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Requested vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:53.066248 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fa0337228d4f8cb6b6c3aab437619e" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:53.066380 9494 raft_consensus.cc:3037] T d4fa0337228d4f8cb6b6c3aab437619e P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.066766 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fa0337228d4f8cb6b6c3aab437619e" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:53.066917 7014 raft_consensus.cc:3037] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.067509 9494 raft_consensus.cc:2454] T d4fa0337228d4f8cb6b6c3aab437619e P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 01:59:53.067808 9559 leader_election.cc:302] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:53.067963 7014 raft_consensus.cc:2454] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 01:59:53.068200 14999 raft_consensus.cc:2781] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:53.068290 14999 raft_consensus.cc:686] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 LEADER]: Becoming Leader. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Running, Role: LEADER
I20240503 01:59:53.068403 14999 consensus_queue.cc:237] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:53.069869 7256 tablet_service.cc:1467] Processing CreateTablet for tablet d8eafdad6586482b98a8cf23ae71ac7c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=eb82d7ba77c6424ca8cdaeb83851ff06]), partition=
I20240503 01:59:53.070148 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8eafdad6586482b98a8cf23ae71ac7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.071218 6732 catalog_manager.cc:5484] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:53.072610 15186 tablet_bootstrap.cc:492] T d8eafdad6586482b98a8cf23ae71ac7c P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:53.074156 9473 tablet_service.cc:1467] Processing CreateTablet for tablet d8eafdad6586482b98a8cf23ae71ac7c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=eb82d7ba77c6424ca8cdaeb83851ff06]), partition=
I20240503 01:59:53.074488 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8eafdad6586482b98a8cf23ae71ac7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.075030 7124 tablet_service.cc:1467] Processing CreateTablet for tablet d8eafdad6586482b98a8cf23ae71ac7c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=eb82d7ba77c6424ca8cdaeb83851ff06]), partition=
I20240503 01:59:53.075359 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8eafdad6586482b98a8cf23ae71ac7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.078153 15141 tablet_bootstrap.cc:492] T d8eafdad6586482b98a8cf23ae71ac7c P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:53.079629 15141 tablet_bootstrap.cc:654] T d8eafdad6586482b98a8cf23ae71ac7c P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.080818 15191 tablet_bootstrap.cc:492] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:53.081393 15141 tablet_bootstrap.cc:492] T d8eafdad6586482b98a8cf23ae71ac7c P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:53.081490 15141 ts_tablet_manager.cc:1367] T d8eafdad6586482b98a8cf23ae71ac7c P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:53.081987 15191 tablet_bootstrap.cc:654] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.081976 15141 raft_consensus.cc:348] T d8eafdad6586482b98a8cf23ae71ac7c P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:53.082132 15141 raft_consensus.cc:374] T d8eafdad6586482b98a8cf23ae71ac7c P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.082175 15141 raft_consensus.cc:729] T d8eafdad6586482b98a8cf23ae71ac7c P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.082298 15141 consensus_queue.cc:260] T d8eafdad6586482b98a8cf23ae71ac7c P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:53.082720 15141 ts_tablet_manager.cc:1398] T d8eafdad6586482b98a8cf23ae71ac7c P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:53.085134 15191 tablet_bootstrap.cc:492] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:53.085247 15191 ts_tablet_manager.cc:1367] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 01:59:53.085772 15191 raft_consensus.cc:348] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:53.085897 15191 raft_consensus.cc:374] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.085927 15191 raft_consensus.cc:729] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.086043 15191 consensus_queue.cc:260] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:53.086323 15191 ts_tablet_manager.cc:1398] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:53.091197 15186 tablet_bootstrap.cc:654] T d8eafdad6586482b98a8cf23ae71ac7c P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.097026 15186 tablet_bootstrap.cc:492] T d8eafdad6586482b98a8cf23ae71ac7c P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:53.097183 15186 ts_tablet_manager.cc:1367] T d8eafdad6586482b98a8cf23ae71ac7c P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20240503 01:59:53.097746 15186 raft_consensus.cc:348] T d8eafdad6586482b98a8cf23ae71ac7c P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:53.097918 15186 raft_consensus.cc:374] T d8eafdad6586482b98a8cf23ae71ac7c P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.097949 15186 raft_consensus.cc:729] T d8eafdad6586482b98a8cf23ae71ac7c P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.098094 15186 consensus_queue.cc:260] T d8eafdad6586482b98a8cf23ae71ac7c P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:53.098704 15186 ts_tablet_manager.cc:1398] T d8eafdad6586482b98a8cf23ae71ac7c P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:53.142503 7014 consensus_queue.cc:237] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:53.144464 7275 raft_consensus.cc:1261] T 96dd9b1f85cf471495dff16d5fa9148a P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:53.144646 6947 consensus_peers.cc:485] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96dd9b1f85cf471495dff16d5fa9148a. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:53.144933 14921 consensus_queue.cc:1035] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:53.145306 7145 raft_consensus.cc:1261] T 96dd9b1f85cf471495dff16d5fa9148a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:53.145737 14921 consensus_queue.cc:1035] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:53.148321 14833 raft_consensus.cc:2932] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:53.148844 7145 raft_consensus.cc:2932] T 96dd9b1f85cf471495dff16d5fa9148a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:53.152458 6732 catalog_manager.cc:5484] T 96dd9b1f85cf471495dff16d5fa9148a P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:53.152861 7275 raft_consensus.cc:2932] T 96dd9b1f85cf471495dff16d5fa9148a P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:53.176113 7014 consensus_queue.cc:237] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:53.177597 6882 raft_consensus.cc:1261] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:53.177976 6949 consensus_peers.cc:485] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c3d43c8b2ca42b19616fde1d7eb47a3. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:53.178067 14833 consensus_queue.cc:1035] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:53.178203 7145 raft_consensus.cc:1261] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:53.178727 14833 consensus_queue.cc:1035] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:53.180883 14243 raft_consensus.cc:2932] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:53.181345 7145 raft_consensus.cc:2932] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:53.183292 6882 raft_consensus.cc:2932] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:53.184722 6737 catalog_manager.cc:5484] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:53.211019 9362 consensus_queue.cc:237] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
W20240503 01:59:53.212972 9298 consensus_peers.cc:485] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8 -> Peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): Couldn't send request to peer c688f3df56ac4b6d9b0c30fa4cc8ca2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 23ae815010364a5586e60b77ff0a600a. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:53.213145 9494 raft_consensus.cc:1261] T 23ae815010364a5586e60b77ff0a600a P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:53.213578 6882 raft_consensus.cc:1261] T 23ae815010364a5586e60b77ff0a600a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:53.213711 13349 consensus_queue.cc:1035] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:53.214731 13349 consensus_queue.cc:1035] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:53.218744 9494 raft_consensus.cc:2932] T 23ae815010364a5586e60b77ff0a600a P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:53.218312 15244 raft_consensus.cc:2932] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:53.222122 6734 catalog_manager.cc:5484] T 23ae815010364a5586e60b77ff0a600a P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:53.223026 6882 raft_consensus.cc:2932] T 23ae815010364a5586e60b77ff0a600a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:53.252254 7014 consensus_queue.cc:237] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
W20240503 01:59:53.254009 6947 consensus_peers.cc:485] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 487eafe47df84b468618f473eca37874. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:53.254001 9362 raft_consensus.cc:1261] T 487eafe47df84b468618f473eca37874 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:53.254036 9757 raft_consensus.cc:1261] T 487eafe47df84b468618f473eca37874 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:53.254406 14243 consensus_queue.cc:1035] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:53.254612 14921 consensus_queue.cc:1035] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:53.256999 14921 raft_consensus.cc:2932] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:53.257478 9757 raft_consensus.cc:2932] T 487eafe47df84b468618f473eca37874 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:53.260286 6737 catalog_manager.cc:5484] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index -1 to 2, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:53.260543 9362 raft_consensus.cc:2932] T 487eafe47df84b468618f473eca37874 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:53.268713 15096 raft_consensus.cc:482] T c0e7400f30244b4785c3389c63c7f9d6 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:53.268868 15096 raft_consensus.cc:504] T c0e7400f30244b4785c3389c63c7f9d6 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:53.269555 15096 leader_election.cc:288] T c0e7400f30244b4785c3389c63c7f9d6 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:53.269954 9362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e7400f30244b4785c3389c63c7f9d6" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:53.269970 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e7400f30244b4785c3389c63c7f9d6" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:53.270084 9362 raft_consensus.cc:2454] T c0e7400f30244b4785c3389c63c7f9d6 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:53.270102 9757 raft_consensus.cc:2454] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:53.270373 9429 leader_election.cc:302] T c0e7400f30244b4785c3389c63c7f9d6 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:53.270713 15096 raft_consensus.cc:2781] T c0e7400f30244b4785c3389c63c7f9d6 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:53.270778 15096 raft_consensus.cc:482] T c0e7400f30244b4785c3389c63c7f9d6 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:53.270835 15096 raft_consensus.cc:3037] T c0e7400f30244b4785c3389c63c7f9d6 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.271869 15096 raft_consensus.cc:504] T c0e7400f30244b4785c3389c63c7f9d6 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:53.272281 15096 leader_election.cc:288] T c0e7400f30244b4785c3389c63c7f9d6 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Requested vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:53.272795 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e7400f30244b4785c3389c63c7f9d6" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:53.272927 9757 raft_consensus.cc:3037] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.272965 9362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e7400f30244b4785c3389c63c7f9d6" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:53.273069 9362 raft_consensus.cc:3037] T c0e7400f30244b4785c3389c63c7f9d6 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.273808 9757 raft_consensus.cc:2454] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:53.274027 9362 raft_consensus.cc:2454] T c0e7400f30244b4785c3389c63c7f9d6 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:53.274684 9429 leader_election.cc:302] T c0e7400f30244b4785c3389c63c7f9d6 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:53.275288 15096 raft_consensus.cc:2781] T c0e7400f30244b4785c3389c63c7f9d6 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:53.275382 15096 raft_consensus.cc:686] T c0e7400f30244b4785c3389c63c7f9d6 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Becoming Leader. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Running, Role: LEADER
I20240503 01:59:53.275524 15096 consensus_queue.cc:237] T c0e7400f30244b4785c3389c63c7f9d6 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:53.281641 6734 catalog_manager.cc:5484] T c0e7400f30244b4785c3389c63c7f9d6 P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: term changed from 0 to 1, leader changed from <none> to bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:53.292336 7014 consensus_queue.cc:237] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
W20240503 01:59:53.294330 6947 consensus_peers.cc:485] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb287629baab4cada193ae57695ef1b5. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:53.294378 9624 raft_consensus.cc:1261] T fb287629baab4cada193ae57695ef1b5 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:53.294411 9757 raft_consensus.cc:1261] T fb287629baab4cada193ae57695ef1b5 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:53.294828 14921 consensus_queue.cc:1035] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:53.295037 14833 consensus_queue.cc:1035] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:53.299918 15164 raft_consensus.cc:482] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:53.300048 15164 raft_consensus.cc:504] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:53.300480 15164 leader_election.cc:288] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:53.300753 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8eafdad6586482b98a8cf23ae71ac7c" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:53.300894 7275 raft_consensus.cc:2454] T d8eafdad6586482b98a8cf23ae71ac7c P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:53.300885 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8eafdad6586482b98a8cf23ae71ac7c" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:53.301029 9494 raft_consensus.cc:2454] T d8eafdad6586482b98a8cf23ae71ac7c P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:53.301180 7079 leader_election.cc:302] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:53.301860 15164 raft_consensus.cc:2781] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:53.301923 15164 raft_consensus.cc:482] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:53.301968 15164 raft_consensus.cc:3037] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.302155 14921 raft_consensus.cc:2932] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:53.302588 9625 raft_consensus.cc:2932] T fb287629baab4cada193ae57695ef1b5 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:53.303508 15164 raft_consensus.cc:504] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:53.303915 15164 leader_election.cc:288] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:53.304333 9757 raft_consensus.cc:2932] T fb287629baab4cada193ae57695ef1b5 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:53.305884 6734 catalog_manager.cc:5484] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index -1 to 2, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:53.310595 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8eafdad6586482b98a8cf23ae71ac7c" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:53.310755 7275 raft_consensus.cc:3037] T d8eafdad6586482b98a8cf23ae71ac7c P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.311304 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8eafdad6586482b98a8cf23ae71ac7c" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:53.311610 9494 raft_consensus.cc:3037] T d8eafdad6586482b98a8cf23ae71ac7c P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.311967 7275 raft_consensus.cc:2454] T d8eafdad6586482b98a8cf23ae71ac7c P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:53.312583 9494 raft_consensus.cc:2454] T d8eafdad6586482b98a8cf23ae71ac7c P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:53.313119 7078 leader_election.cc:302] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:53.313550 15163 raft_consensus.cc:2781] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:53.313666 15163 raft_consensus.cc:686] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Becoming Leader. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Running, Role: LEADER
I20240503 01:59:53.313812 15163 consensus_queue.cc:237] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:53.319852 6737 catalog_manager.cc:5484] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:53.394243 13671 consensus_queue.cc:1035] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 01:59:53.413553 15096 consensus_queue.cc:1035] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:53.428807 6738 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:53.439033 7275 raft_consensus.cc:1261] T d8eafdad6586482b98a8cf23ae71ac7c P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:53.439687 9494 raft_consensus.cc:1261] T d8eafdad6586482b98a8cf23ae71ac7c P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:53.440088 15159 consensus_queue.cc:1035] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:53.440353 15164 consensus_queue.cc:1035] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:53.446842 6735 catalog_manager.cc:5576] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471) does not have the latest schema for tablet d8eafdad6586482b98a8cf23ae71ac7c (table rebalancer_dynamic_table [id=eb82d7ba77c6424ca8cdaeb83851ff06]). Expected version 1 got 0
I20240503 01:59:53.450387 15337 tablet.cc:1722] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:53.450873 15337 tablet.cc:1620] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: MemRowSet was empty: no flush needed.
I20240503 01:59:53.453256 6734 catalog_manager.cc:5576] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271) does not have the latest schema for tablet d8eafdad6586482b98a8cf23ae71ac7c (table rebalancer_dynamic_table [id=eb82d7ba77c6424ca8cdaeb83851ff06]). Expected version 1 got 0
I20240503 01:59:53.457903 6734 catalog_manager.cc:5576] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271) does not have the latest schema for tablet d8eafdad6586482b98a8cf23ae71ac7c (table rebalancer_dynamic_table [id=eb82d7ba77c6424ca8cdaeb83851ff06]). Expected version 1 got 0
I20240503 01:59:53.458576 15338 tablet.cc:1722] T d8eafdad6586482b98a8cf23ae71ac7c P 1d0e738c34af40fa9950c6f086619b3a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:53.459173 6739 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=eb82d7ba77c6424ca8cdaeb83851ff06] alter complete (version 1)
I20240503 01:59:53.460093 15338 tablet.cc:1620] T d8eafdad6586482b98a8cf23ae71ac7c P 1d0e738c34af40fa9950c6f086619b3a: MemRowSet was empty: no flush needed.
I20240503 01:59:53.463551 6737 catalog_manager.cc:5576] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471) does not have the latest schema for tablet d8eafdad6586482b98a8cf23ae71ac7c (table rebalancer_dynamic_table [id=eb82d7ba77c6424ca8cdaeb83851ff06]). Expected version 1 got 0
I20240503 01:59:53.464238 15339 tablet.cc:1722] T d8eafdad6586482b98a8cf23ae71ac7c P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:53.464843 15350 ts_tablet_manager.cc:902] T dcc773c6d5794e26a0003c7d4acc5d21 P 12e8c683db054df3b75c6285b98a5ad8: Initiating tablet copy from peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:53.464813 15339 tablet.cc:1620] T d8eafdad6586482b98a8cf23ae71ac7c P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:53.465541 15350 tablet_copy_client.cc:308] T dcc773c6d5794e26a0003c7d4acc5d21 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:42943
I20240503 01:59:53.466229 6903 tablet_copy_service.cc:140] P 5dcc33cc621a4cbbae155f3230fb7c46: Received BeginTabletCopySession request for tablet dcc773c6d5794e26a0003c7d4acc5d21 from peer 12e8c683db054df3b75c6285b98a5ad8 ({username='slave'} at 127.0.99.133:55147)
I20240503 01:59:53.466338 6903 tablet_copy_service.cc:161] P 5dcc33cc621a4cbbae155f3230fb7c46: Beginning new tablet copy session on tablet dcc773c6d5794e26a0003c7d4acc5d21 from peer 12e8c683db054df3b75c6285b98a5ad8 at {username='slave'} at 127.0.99.133:55147: session id = 12e8c683db054df3b75c6285b98a5ad8-dcc773c6d5794e26a0003c7d4acc5d21
I20240503 01:59:53.467394 6903 tablet_copy_source_session.cc:205] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:53.474695 15350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc773c6d5794e26a0003c7d4acc5d21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.477399 7256 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:53.478418 15350 tablet_copy_client.cc:783] T dcc773c6d5794e26a0003c7d4acc5d21 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:53.478688 15350 tablet_copy_client.cc:647] T dcc773c6d5794e26a0003c7d4acc5d21 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:53.483361 9341 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:53.485648 15350 tablet_copy_client.cc:523] T dcc773c6d5794e26a0003c7d4acc5d21 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:53.487486 15350 tablet_bootstrap.cc:492] T dcc773c6d5794e26a0003c7d4acc5d21 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:53.498258 9473 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:53.499680 6863 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:53.503762 6993 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:53.532505 15361 ts_tablet_manager.cc:902] T 822bafcbc6de406997955ff7a1d9b612 P bb5364d730cb4489a0bfc6c270b7450a: Initiating tablet copy from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:53.533174 15361 tablet_copy_client.cc:308] T 822bafcbc6de406997955ff7a1d9b612 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41979
I20240503 01:59:53.533739 7165 tablet_copy_service.cc:140] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Received BeginTabletCopySession request for tablet 822bafcbc6de406997955ff7a1d9b612 from peer bb5364d730cb4489a0bfc6c270b7450a ({username='slave'} at 127.0.99.134:59853)
I20240503 01:59:53.533865 7165 tablet_copy_service.cc:161] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Beginning new tablet copy session on tablet 822bafcbc6de406997955ff7a1d9b612 from peer bb5364d730cb4489a0bfc6c270b7450a at {username='slave'} at 127.0.99.134:59853: session id = bb5364d730cb4489a0bfc6c270b7450a-822bafcbc6de406997955ff7a1d9b612
I20240503 01:59:53.535061 7165 tablet_copy_source_session.cc:205] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:53.537051 7124 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:53.538066 15361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 822bafcbc6de406997955ff7a1d9b612. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.544421 9604 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:53.555048 15361 tablet_copy_client.cc:783] T 822bafcbc6de406997955ff7a1d9b612 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:53.555397 15361 tablet_copy_client.cc:647] T 822bafcbc6de406997955ff7a1d9b612 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:53.558589 15361 tablet_copy_client.cc:523] T 822bafcbc6de406997955ff7a1d9b612 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:53.561693 9736 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:53.561864 15361 tablet_bootstrap.cc:492] T 822bafcbc6de406997955ff7a1d9b612 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:53.563571 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:53.570817 15337 tablet.cc:1722] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:53.571171 15337 tablet.cc:1620] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: MemRowSet was empty: no flush needed.
I20240503 01:59:53.572510 15339 tablet.cc:1722] T d8eafdad6586482b98a8cf23ae71ac7c P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:53.572839 15339 tablet.cc:1620] T d8eafdad6586482b98a8cf23ae71ac7c P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:53.574842 15338 tablet.cc:1722] T d8eafdad6586482b98a8cf23ae71ac7c P 1d0e738c34af40fa9950c6f086619b3a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:53.575160 15338 tablet.cc:1620] T d8eafdad6586482b98a8cf23ae71ac7c P 1d0e738c34af40fa9950c6f086619b3a: MemRowSet was empty: no flush needed.
I20240503 01:59:53.578699 15350 tablet_bootstrap.cc:492] T dcc773c6d5794e26a0003c7d4acc5d21 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:53.580410 6734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=eb82d7ba77c6424ca8cdaeb83851ff06] alter complete (version 2)
I20240503 01:59:53.604080 15366 ts_tablet_manager.cc:902] T 96dd9b1f85cf471495dff16d5fa9148a P bb5364d730cb4489a0bfc6c270b7450a: Initiating tablet copy from peer 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:53.605085 15361 tablet_bootstrap.cc:492] T 822bafcbc6de406997955ff7a1d9b612 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:53.605321 15366 tablet_copy_client.cc:308] T 96dd9b1f85cf471495dff16d5fa9148a P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:37231
I20240503 01:59:53.606139 7034 tablet_copy_service.cc:140] P 63331f455be049c1bdd6c23c59ac1bef: Received BeginTabletCopySession request for tablet 96dd9b1f85cf471495dff16d5fa9148a from peer bb5364d730cb4489a0bfc6c270b7450a ({username='slave'} at 127.0.99.134:41551)
I20240503 01:59:53.606261 7034 tablet_copy_service.cc:161] P 63331f455be049c1bdd6c23c59ac1bef: Beginning new tablet copy session on tablet 96dd9b1f85cf471495dff16d5fa9148a from peer bb5364d730cb4489a0bfc6c270b7450a at {username='slave'} at 127.0.99.134:41551: session id = bb5364d730cb4489a0bfc6c270b7450a-96dd9b1f85cf471495dff16d5fa9148a
I20240503 01:59:53.607313 7034 tablet_copy_source_session.cc:205] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:53.608397 14749 consensus_queue.cc:1035] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:53.610368 15366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96dd9b1f85cf471495dff16d5fa9148a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.610818 15350 tablet_bootstrap.cc:492] T dcc773c6d5794e26a0003c7d4acc5d21 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap complete.
I20240503 01:59:53.611109 15350 ts_tablet_manager.cc:1367] T dcc773c6d5794e26a0003c7d4acc5d21 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.124s user 0.029s sys 0.000s
I20240503 01:59:53.611675 15350 raft_consensus.cc:348] T dcc773c6d5794e26a0003c7d4acc5d21 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:53.611805 15350 raft_consensus.cc:729] T dcc773c6d5794e26a0003c7d4acc5d21 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.611951 15350 consensus_queue.cc:260] T dcc773c6d5794e26a0003c7d4acc5d21 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:53.612294 15350 ts_tablet_manager.cc:1398] T dcc773c6d5794e26a0003c7d4acc5d21 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:53.613224 6903 tablet_copy_service.cc:342] P 5dcc33cc621a4cbbae155f3230fb7c46: Request end of tablet copy session 12e8c683db054df3b75c6285b98a5ad8-dcc773c6d5794e26a0003c7d4acc5d21 received from {username='slave'} at 127.0.99.133:55147
I20240503 01:59:53.613319 6903 tablet_copy_service.cc:433] P 5dcc33cc621a4cbbae155f3230fb7c46: ending tablet copy session 12e8c683db054df3b75c6285b98a5ad8-dcc773c6d5794e26a0003c7d4acc5d21 on tablet dcc773c6d5794e26a0003c7d4acc5d21 with peer 12e8c683db054df3b75c6285b98a5ad8
I20240503 01:59:53.613857 14749 consensus_queue.cc:1035] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:53.621476 15378 ts_tablet_manager.cc:902] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Initiating tablet copy from peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:53.622155 15378 tablet_copy_client.cc:308] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:37387
I20240503 01:59:53.622596 9383 tablet_copy_service.cc:140] P 12e8c683db054df3b75c6285b98a5ad8: Received BeginTabletCopySession request for tablet 23ae815010364a5586e60b77ff0a600a from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a ({username='slave'} at 127.0.99.131:49995)
I20240503 01:59:53.622699 9383 tablet_copy_service.cc:161] P 12e8c683db054df3b75c6285b98a5ad8: Beginning new tablet copy session on tablet 23ae815010364a5586e60b77ff0a600a from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a at {username='slave'} at 127.0.99.131:49995: session id = c688f3df56ac4b6d9b0c30fa4cc8ca2a-23ae815010364a5586e60b77ff0a600a
I20240503 01:59:53.623709 9383 tablet_copy_source_session.cc:205] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:53.624971 15361 tablet_bootstrap.cc:492] T 822bafcbc6de406997955ff7a1d9b612 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap complete.
I20240503 01:59:53.625272 15361 ts_tablet_manager.cc:1367] T 822bafcbc6de406997955ff7a1d9b612 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.063s user 0.026s sys 0.000s
I20240503 01:59:53.625815 15361 raft_consensus.cc:348] T 822bafcbc6de406997955ff7a1d9b612 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:53.626003 15361 raft_consensus.cc:729] T 822bafcbc6de406997955ff7a1d9b612 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.626200 15361 consensus_queue.cc:260] T 822bafcbc6de406997955ff7a1d9b612 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:53.627328 15361 ts_tablet_manager.cc:1398] T 822bafcbc6de406997955ff7a1d9b612 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:53.627555 15366 tablet_copy_client.cc:783] T 96dd9b1f85cf471495dff16d5fa9148a P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:53.627701 15366 tablet_copy_client.cc:647] T 96dd9b1f85cf471495dff16d5fa9148a P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:53.627777 7165 tablet_copy_service.cc:342] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Request end of tablet copy session bb5364d730cb4489a0bfc6c270b7450a-822bafcbc6de406997955ff7a1d9b612 received from {username='slave'} at 127.0.99.134:59853
I20240503 01:59:53.627852 7165 tablet_copy_service.cc:433] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: ending tablet copy session bb5364d730cb4489a0bfc6c270b7450a-822bafcbc6de406997955ff7a1d9b612 on tablet 822bafcbc6de406997955ff7a1d9b612 with peer bb5364d730cb4489a0bfc6c270b7450a
I20240503 01:59:53.630100 15368 ts_tablet_manager.cc:902] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P b7062fbbc2b54fff9e99ec4ea5562c76: Initiating tablet copy from peer 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:53.630702 15368 tablet_copy_client.cc:308] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:37231
I20240503 01:59:53.630714 15366 tablet_copy_client.cc:523] T 96dd9b1f85cf471495dff16d5fa9148a P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:53.632211 15366 tablet_bootstrap.cc:492] T 96dd9b1f85cf471495dff16d5fa9148a P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:53.632206 7034 tablet_copy_service.cc:140] P 63331f455be049c1bdd6c23c59ac1bef: Received BeginTabletCopySession request for tablet 8c3d43c8b2ca42b19616fde1d7eb47a3 from peer b7062fbbc2b54fff9e99ec4ea5562c76 ({username='slave'} at 127.0.99.135:36345)
I20240503 01:59:53.632333 7034 tablet_copy_service.cc:161] P 63331f455be049c1bdd6c23c59ac1bef: Beginning new tablet copy session on tablet 8c3d43c8b2ca42b19616fde1d7eb47a3 from peer b7062fbbc2b54fff9e99ec4ea5562c76 at {username='slave'} at 127.0.99.135:36345: session id = b7062fbbc2b54fff9e99ec4ea5562c76-8c3d43c8b2ca42b19616fde1d7eb47a3
I20240503 01:59:53.633342 7034 tablet_copy_source_session.cc:205] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:53.634608 15378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23ae815010364a5586e60b77ff0a600a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.637179 15378 tablet_copy_client.cc:783] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:53.637404 15378 tablet_copy_client.cc:647] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:53.639047 15368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c3d43c8b2ca42b19616fde1d7eb47a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.642277 15368 tablet_copy_client.cc:783] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:53.642560 15368 tablet_copy_client.cc:647] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:53.646117 15368 tablet_copy_client.cc:523] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:53.647716 15368 tablet_bootstrap.cc:492] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:53.647985 15378 tablet_copy_client.cc:523] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:53.650810 15378 tablet_bootstrap.cc:492] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:53.658218 15378 tablet_bootstrap.cc:492] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:53.658751 15378 tablet_bootstrap.cc:492] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap complete.
I20240503 01:59:53.659152 15378 ts_tablet_manager.cc:1367] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20240503 01:59:53.665079 15378 raft_consensus.cc:348] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:53.665380 15378 raft_consensus.cc:729] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.665594 15378 consensus_queue.cc:260] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:53.666172 15378 ts_tablet_manager.cc:1398] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 01:59:53.667281 9383 tablet_copy_service.cc:342] P 12e8c683db054df3b75c6285b98a5ad8: Request end of tablet copy session c688f3df56ac4b6d9b0c30fa4cc8ca2a-23ae815010364a5586e60b77ff0a600a received from {username='slave'} at 127.0.99.131:49995
I20240503 01:59:53.667371 9383 tablet_copy_service.cc:433] P 12e8c683db054df3b75c6285b98a5ad8: ending tablet copy session c688f3df56ac4b6d9b0c30fa4cc8ca2a-23ae815010364a5586e60b77ff0a600a on tablet 23ae815010364a5586e60b77ff0a600a with peer c688f3df56ac4b6d9b0c30fa4cc8ca2a
I20240503 01:59:53.673411 15368 tablet_bootstrap.cc:492] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:53.673749 15368 tablet_bootstrap.cc:492] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap complete.
I20240503 01:59:53.673875 15366 tablet_bootstrap.cc:492] T 96dd9b1f85cf471495dff16d5fa9148a P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:53.673938 15368 ts_tablet_manager.cc:1367] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.026s user 0.021s sys 0.002s
I20240503 01:59:53.674140 15366 tablet_bootstrap.cc:492] T 96dd9b1f85cf471495dff16d5fa9148a P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap complete.
I20240503 01:59:53.674301 15366 ts_tablet_manager.cc:1367] T 96dd9b1f85cf471495dff16d5fa9148a P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.042s user 0.029s sys 0.000s
I20240503 01:59:53.674453 15368 raft_consensus.cc:348] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:53.674664 15368 raft_consensus.cc:729] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.674772 15366 raft_consensus.cc:348] T 96dd9b1f85cf471495dff16d5fa9148a P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:53.674871 15368 consensus_queue.cc:260] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:53.674914 15366 raft_consensus.cc:729] T 96dd9b1f85cf471495dff16d5fa9148a P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.675099 15366 consensus_queue.cc:260] T 96dd9b1f85cf471495dff16d5fa9148a P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:53.675225 15368 ts_tablet_manager.cc:1398] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 01:59:53.675427 15366 ts_tablet_manager.cc:1398] T 96dd9b1f85cf471495dff16d5fa9148a P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 01:59:53.675944 7034 tablet_copy_service.cc:342] P 63331f455be049c1bdd6c23c59ac1bef: Request end of tablet copy session bb5364d730cb4489a0bfc6c270b7450a-96dd9b1f85cf471495dff16d5fa9148a received from {username='slave'} at 127.0.99.134:41551
I20240503 01:59:53.676023 7034 tablet_copy_service.cc:433] P 63331f455be049c1bdd6c23c59ac1bef: ending tablet copy session bb5364d730cb4489a0bfc6c270b7450a-96dd9b1f85cf471495dff16d5fa9148a on tablet 96dd9b1f85cf471495dff16d5fa9148a with peer bb5364d730cb4489a0bfc6c270b7450a
I20240503 01:59:53.677029 7034 tablet_copy_service.cc:342] P 63331f455be049c1bdd6c23c59ac1bef: Request end of tablet copy session b7062fbbc2b54fff9e99ec4ea5562c76-8c3d43c8b2ca42b19616fde1d7eb47a3 received from {username='slave'} at 127.0.99.135:36345
I20240503 01:59:53.677124 7034 tablet_copy_service.cc:433] P 63331f455be049c1bdd6c23c59ac1bef: ending tablet copy session b7062fbbc2b54fff9e99ec4ea5562c76-8c3d43c8b2ca42b19616fde1d7eb47a3 on tablet 8c3d43c8b2ca42b19616fde1d7eb47a3 with peer b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 01:59:53.686400 6739 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:53.693588 15337 tablet.cc:1722] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:53.693761 15338 tablet.cc:1722] T d8eafdad6586482b98a8cf23ae71ac7c P 1d0e738c34af40fa9950c6f086619b3a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:53.699230 15339 tablet.cc:1722] T d8eafdad6586482b98a8cf23ae71ac7c P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:53.703472 6737 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=eb82d7ba77c6424ca8cdaeb83851ff06] alter complete (version 3)
I20240503 01:59:53.732537 15366 ts_tablet_manager.cc:902] T 487eafe47df84b468618f473eca37874 P bb5364d730cb4489a0bfc6c270b7450a: Initiating tablet copy from peer 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:53.733068 15366 tablet_copy_client.cc:308] T 487eafe47df84b468618f473eca37874 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:37231
I20240503 01:59:53.733647 7034 tablet_copy_service.cc:140] P 63331f455be049c1bdd6c23c59ac1bef: Received BeginTabletCopySession request for tablet 487eafe47df84b468618f473eca37874 from peer bb5364d730cb4489a0bfc6c270b7450a ({username='slave'} at 127.0.99.134:41551)
I20240503 01:59:53.733778 7034 tablet_copy_service.cc:161] P 63331f455be049c1bdd6c23c59ac1bef: Beginning new tablet copy session on tablet 487eafe47df84b468618f473eca37874 from peer bb5364d730cb4489a0bfc6c270b7450a at {username='slave'} at 127.0.99.134:41551: session id = bb5364d730cb4489a0bfc6c270b7450a-487eafe47df84b468618f473eca37874
I20240503 01:59:53.734901 7034 tablet_copy_source_session.cc:205] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:53.737596 15366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 487eafe47df84b468618f473eca37874. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.740115 15366 tablet_copy_client.cc:783] T 487eafe47df84b468618f473eca37874 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:53.741338 15366 tablet_copy_client.cc:647] T 487eafe47df84b468618f473eca37874 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:53.747427 15366 tablet_copy_client.cc:523] T 487eafe47df84b468618f473eca37874 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:53.749926 15366 tablet_bootstrap.cc:492] T 487eafe47df84b468618f473eca37874 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:53.750613 15361 ts_tablet_manager.cc:902] T fb287629baab4cada193ae57695ef1b5 P bb5364d730cb4489a0bfc6c270b7450a: Initiating tablet copy from peer 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:53.751212 15361 tablet_copy_client.cc:308] T fb287629baab4cada193ae57695ef1b5 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:37231
I20240503 01:59:53.751314 15330 consensus_queue.cc:1035] T c0e7400f30244b4785c3389c63c7f9d6 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:53.751708 7034 tablet_copy_service.cc:140] P 63331f455be049c1bdd6c23c59ac1bef: Received BeginTabletCopySession request for tablet fb287629baab4cada193ae57695ef1b5 from peer bb5364d730cb4489a0bfc6c270b7450a ({username='slave'} at 127.0.99.134:41551)
I20240503 01:59:53.751832 7034 tablet_copy_service.cc:161] P 63331f455be049c1bdd6c23c59ac1bef: Beginning new tablet copy session on tablet fb287629baab4cada193ae57695ef1b5 from peer bb5364d730cb4489a0bfc6c270b7450a at {username='slave'} at 127.0.99.134:41551: session id = bb5364d730cb4489a0bfc6c270b7450a-fb287629baab4cada193ae57695ef1b5
I20240503 01:59:53.752799 7034 tablet_copy_source_session.cc:205] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:53.755043 15366 tablet_bootstrap.cc:492] T 487eafe47df84b468618f473eca37874 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:53.755319 15366 tablet_bootstrap.cc:492] T 487eafe47df84b468618f473eca37874 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap complete.
I20240503 01:59:53.755470 15366 ts_tablet_manager.cc:1367] T 487eafe47df84b468618f473eca37874 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240503 01:59:53.755903 15366 raft_consensus.cc:348] T 487eafe47df84b468618f473eca37874 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:53.756017 15366 raft_consensus.cc:729] T 487eafe47df84b468618f473eca37874 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.756150 15366 consensus_queue.cc:260] T 487eafe47df84b468618f473eca37874 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:53.756421 15366 ts_tablet_manager.cc:1398] T 487eafe47df84b468618f473eca37874 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 01:59:53.756774 15361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb287629baab4cada193ae57695ef1b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.759393 15361 tablet_copy_client.cc:783] T fb287629baab4cada193ae57695ef1b5 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:53.759559 15361 tablet_copy_client.cc:647] T fb287629baab4cada193ae57695ef1b5 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:53.760097 15330 consensus_queue.cc:1035] T c0e7400f30244b4785c3389c63c7f9d6 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:53.760215 7034 tablet_copy_service.cc:342] P 63331f455be049c1bdd6c23c59ac1bef: Request end of tablet copy session bb5364d730cb4489a0bfc6c270b7450a-487eafe47df84b468618f473eca37874 received from {username='slave'} at 127.0.99.134:41551
I20240503 01:59:53.760303 7034 tablet_copy_service.cc:433] P 63331f455be049c1bdd6c23c59ac1bef: ending tablet copy session bb5364d730cb4489a0bfc6c270b7450a-487eafe47df84b468618f473eca37874 on tablet 487eafe47df84b468618f473eca37874 with peer bb5364d730cb4489a0bfc6c270b7450a
I20240503 01:59:53.762672 15361 tablet_copy_client.cc:523] T fb287629baab4cada193ae57695ef1b5 P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:53.765780 15361 tablet_bootstrap.cc:492] T fb287629baab4cada193ae57695ef1b5 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:53.769358 15361 tablet_bootstrap.cc:492] T fb287629baab4cada193ae57695ef1b5 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:53.769610 15361 tablet_bootstrap.cc:492] T fb287629baab4cada193ae57695ef1b5 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap complete.
I20240503 01:59:53.769771 15361 ts_tablet_manager.cc:1367] T fb287629baab4cada193ae57695ef1b5 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 01:59:53.770246 15361 raft_consensus.cc:348] T fb287629baab4cada193ae57695ef1b5 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:53.770401 15361 raft_consensus.cc:729] T fb287629baab4cada193ae57695ef1b5 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.770587 15361 consensus_queue.cc:260] T fb287629baab4cada193ae57695ef1b5 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:53.770956 15361 ts_tablet_manager.cc:1398] T fb287629baab4cada193ae57695ef1b5 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 01:59:53.771632 7034 tablet_copy_service.cc:342] P 63331f455be049c1bdd6c23c59ac1bef: Request end of tablet copy session bb5364d730cb4489a0bfc6c270b7450a-fb287629baab4cada193ae57695ef1b5 received from {username='slave'} at 127.0.99.134:41551
I20240503 01:59:53.771725 7034 tablet_copy_service.cc:433] P 63331f455be049c1bdd6c23c59ac1bef: ending tablet copy session bb5364d730cb4489a0bfc6c270b7450a-fb287629baab4cada193ae57695ef1b5 on tablet fb287629baab4cada193ae57695ef1b5 with peer bb5364d730cb4489a0bfc6c270b7450a
I20240503 01:59:53.806860 6734 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:53.807118 6734 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:53.810094 6734 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet d8eafdad6586482b98a8cf23ae71ac7c
I20240503 01:59:53.810766 7256 tablet_service.cc:1514] Processing DeleteTablet for tablet d8eafdad6586482b98a8cf23ae71ac7c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:53 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:53.811177 7124 tablet_service.cc:1514] Processing DeleteTablet for tablet d8eafdad6586482b98a8cf23ae71ac7c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:53 UTC) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:53.811515 15459 tablet_replica.cc:330] T d8eafdad6586482b98a8cf23ae71ac7c P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:53.811686 9473 tablet_service.cc:1514] Processing DeleteTablet for tablet d8eafdad6586482b98a8cf23ae71ac7c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:53 UTC) from {username='slave'} at 127.0.0.1:54678
I20240503 01:59:53.812021 15459 raft_consensus.cc:2229] T d8eafdad6586482b98a8cf23ae71ac7c P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:53.812310 15459 raft_consensus.cc:2258] T d8eafdad6586482b98a8cf23ae71ac7c P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:53.813011 15459 ts_tablet_manager.cc:1875] T d8eafdad6586482b98a8cf23ae71ac7c P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:53.815584 15461 tablet_replica.cc:330] T d8eafdad6586482b98a8cf23ae71ac7c P bb5364d730cb4489a0bfc6c270b7450a: stopping tablet replica
I20240503 01:59:53.816134 15459 ts_tablet_manager.cc:1888] T d8eafdad6586482b98a8cf23ae71ac7c P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:53.816181 15460 tablet_replica.cc:330] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:53.816246 15459 log.cc:1196] T d8eafdad6586482b98a8cf23ae71ac7c P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/d8eafdad6586482b98a8cf23ae71ac7c
I20240503 01:59:53.816594 15460 raft_consensus.cc:2229] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:53.816927 15459 ts_tablet_manager.cc:1909] T d8eafdad6586482b98a8cf23ae71ac7c P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:53.816942 15460 raft_consensus.cc:2258] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:53.817916 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet d8eafdad6586482b98a8cf23ae71ac7c (table rebalancer_dynamic_table_new_name [id=eb82d7ba77c6424ca8cdaeb83851ff06]) successfully deleted
I20240503 01:59:53.818357 15461 raft_consensus.cc:2229] T d8eafdad6586482b98a8cf23ae71ac7c P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:53.819017 15461 raft_consensus.cc:2258] T d8eafdad6586482b98a8cf23ae71ac7c P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:53.817680 15460 ts_tablet_manager.cc:1875] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:53.820451 15461 ts_tablet_manager.cc:1875] T d8eafdad6586482b98a8cf23ae71ac7c P bb5364d730cb4489a0bfc6c270b7450a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:53.823779 15460 ts_tablet_manager.cc:1888] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:53.824019 15460 log.cc:1196] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/d8eafdad6586482b98a8cf23ae71ac7c
I20240503 01:59:53.824465 15460 ts_tablet_manager.cc:1909] T d8eafdad6586482b98a8cf23ae71ac7c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting consensus metadata
I20240503 01:59:53.824779 15461 ts_tablet_manager.cc:1888] T d8eafdad6586482b98a8cf23ae71ac7c P bb5364d730cb4489a0bfc6c270b7450a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:53.824918 15461 log.cc:1196] T d8eafdad6586482b98a8cf23ae71ac7c P bb5364d730cb4489a0bfc6c270b7450a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/d8eafdad6586482b98a8cf23ae71ac7c
I20240503 01:59:53.825449 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet d8eafdad6586482b98a8cf23ae71ac7c (table rebalancer_dynamic_table_new_name [id=eb82d7ba77c6424ca8cdaeb83851ff06]) successfully deleted
I20240503 01:59:53.826078 15461 ts_tablet_manager.cc:1909] T d8eafdad6586482b98a8cf23ae71ac7c P bb5364d730cb4489a0bfc6c270b7450a: Deleting consensus metadata
I20240503 01:59:53.826992 6725 catalog_manager.cc:4830] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): tablet d8eafdad6586482b98a8cf23ae71ac7c (table rebalancer_dynamic_table_new_name [id=eb82d7ba77c6424ca8cdaeb83851ff06]) successfully deleted
I20240503 01:59:53.835639 7256 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:53.841596 6863 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:53.842729 9341 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:53.848096 6993 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:53.848829 6734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:53.859294 9736 tablet_service.cc:1467] Processing CreateTablet for tablet f8d3d130f4a44904924f7ccf4316fc75 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=1de39c3b40b04d1483699c7fc0d0c6cb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:53.859285 9604 tablet_service.cc:1467] Processing CreateTablet for tablet d80398c9a264430197262bb42241d58d (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=1de39c3b40b04d1483699c7fc0d0c6cb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:53.859668 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d80398c9a264430197262bb42241d58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.859692 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8d3d130f4a44904924f7ccf4316fc75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.862613 9735 tablet_service.cc:1467] Processing CreateTablet for tablet dccce42f6cd64934808d8c6e5f366e35 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=1de39c3b40b04d1483699c7fc0d0c6cb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:53.862900 9735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dccce42f6cd64934808d8c6e5f366e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.862799 6861 tablet_service.cc:1467] Processing CreateTablet for tablet dccce42f6cd64934808d8c6e5f366e35 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=1de39c3b40b04d1483699c7fc0d0c6cb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:53.862799 6863 tablet_service.cc:1467] Processing CreateTablet for tablet f8d3d130f4a44904924f7ccf4316fc75 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=1de39c3b40b04d1483699c7fc0d0c6cb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:53.863145 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8d3d130f4a44904924f7ccf4316fc75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.865301 15475 tablet_bootstrap.cc:492] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:53.865588 7124 tablet_service.cc:1467] Processing CreateTablet for tablet d80398c9a264430197262bb42241d58d (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=1de39c3b40b04d1483699c7fc0d0c6cb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:53.865890 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d80398c9a264430197262bb42241d58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.866890 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dccce42f6cd64934808d8c6e5f366e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.867728 15473 tablet_bootstrap.cc:492] T d80398c9a264430197262bb42241d58d P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:53.868659 9341 tablet_service.cc:1467] Processing CreateTablet for tablet f8d3d130f4a44904924f7ccf4316fc75 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=1de39c3b40b04d1483699c7fc0d0c6cb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:53.868757 15473 tablet_bootstrap.cc:654] T d80398c9a264430197262bb42241d58d P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.868955 9341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8d3d130f4a44904924f7ccf4316fc75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.869112 6993 tablet_service.cc:1467] Processing CreateTablet for tablet d80398c9a264430197262bb42241d58d (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=1de39c3b40b04d1483699c7fc0d0c6cb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:53.869369 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d80398c9a264430197262bb42241d58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.870041 15475 tablet_bootstrap.cc:654] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.871361 15479 tablet_bootstrap.cc:492] T f8d3d130f4a44904924f7ccf4316fc75 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:53.872318 9604 tablet_service.cc:1467] Processing CreateTablet for tablet dccce42f6cd64934808d8c6e5f366e35 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=1de39c3b40b04d1483699c7fc0d0c6cb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:53.872581 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dccce42f6cd64934808d8c6e5f366e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.872596 15479 tablet_bootstrap.cc:654] T f8d3d130f4a44904924f7ccf4316fc75 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.875785 9604 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 01:59:53.877291 9473 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:53.878366 15480 tablet_bootstrap.cc:492] T d80398c9a264430197262bb42241d58d P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:53.880187 15480 tablet_bootstrap.cc:654] T d80398c9a264430197262bb42241d58d P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.880723 15476 tablet_bootstrap.cc:492] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:53.881282 15474 tablet_bootstrap.cc:492] T f8d3d130f4a44904924f7ccf4316fc75 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:53.881904 15476 tablet_bootstrap.cc:654] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.882870 15473 tablet_bootstrap.cc:492] T d80398c9a264430197262bb42241d58d P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:53.883023 15473 ts_tablet_manager.cc:1367] T d80398c9a264430197262bb42241d58d P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.001s
I20240503 01:59:53.883213 15475 tablet_bootstrap.cc:492] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:53.883317 15475 ts_tablet_manager.cc:1367] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240503 01:59:53.884214 15475 raft_consensus.cc:348] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:53.884505 15475 raft_consensus.cc:374] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.884579 15475 raft_consensus.cc:729] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.884694 15473 raft_consensus.cc:348] T d80398c9a264430197262bb42241d58d P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:53.884810 15473 raft_consensus.cc:374] T d80398c9a264430197262bb42241d58d P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.884743 15475 consensus_queue.cc:260] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:53.884840 15473 raft_consensus.cc:729] T d80398c9a264430197262bb42241d58d P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.884968 15473 consensus_queue.cc:260] T d80398c9a264430197262bb42241d58d P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:53.885079 15475 ts_tablet_manager.cc:1398] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:53.885277 15475 tablet_bootstrap.cc:492] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:53.886385 15473 ts_tablet_manager.cc:1398] T d80398c9a264430197262bb42241d58d P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:53.886564 15473 tablet_bootstrap.cc:492] T dccce42f6cd64934808d8c6e5f366e35 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:53.888319 15474 tablet_bootstrap.cc:654] T f8d3d130f4a44904924f7ccf4316fc75 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.889307 15473 tablet_bootstrap.cc:654] T dccce42f6cd64934808d8c6e5f366e35 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.890825 15480 tablet_bootstrap.cc:492] T d80398c9a264430197262bb42241d58d P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:53.890933 15480 ts_tablet_manager.cc:1367] T d80398c9a264430197262bb42241d58d P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240503 01:59:53.891530 15480 raft_consensus.cc:348] T d80398c9a264430197262bb42241d58d P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:53.891889 15480 raft_consensus.cc:374] T d80398c9a264430197262bb42241d58d P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.891924 15480 raft_consensus.cc:729] T d80398c9a264430197262bb42241d58d P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.892072 15480 consensus_queue.cc:260] T d80398c9a264430197262bb42241d58d P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:53.892414 15480 ts_tablet_manager.cc:1398] T d80398c9a264430197262bb42241d58d P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:53.895740 15475 tablet_bootstrap.cc:654] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.902385 9735 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:53.907976 7124 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:53.913151 6734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:53.921893 15474 tablet_bootstrap.cc:492] T f8d3d130f4a44904924f7ccf4316fc75 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:53.921981 15473 tablet_bootstrap.cc:492] T dccce42f6cd64934808d8c6e5f366e35 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:53.922024 15474 ts_tablet_manager.cc:1367] T f8d3d130f4a44904924f7ccf4316fc75 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.000s
I20240503 01:59:53.922063 15473 ts_tablet_manager.cc:1367] T dccce42f6cd64934808d8c6e5f366e35 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.036s user 0.001s sys 0.001s
I20240503 01:59:53.922672 15473 raft_consensus.cc:348] T dccce42f6cd64934808d8c6e5f366e35 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:53.922669 15474 raft_consensus.cc:348] T f8d3d130f4a44904924f7ccf4316fc75 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:53.922784 15473 raft_consensus.cc:374] T dccce42f6cd64934808d8c6e5f366e35 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.922784 15474 raft_consensus.cc:374] T f8d3d130f4a44904924f7ccf4316fc75 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.922819 15474 raft_consensus.cc:729] T f8d3d130f4a44904924f7ccf4316fc75 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.922819 15473 raft_consensus.cc:729] T dccce42f6cd64934808d8c6e5f366e35 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.922947 15473 consensus_queue.cc:260] T dccce42f6cd64934808d8c6e5f366e35 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:53.922947 15474 consensus_queue.cc:260] T f8d3d130f4a44904924f7ccf4316fc75 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:53.923243 15473 ts_tablet_manager.cc:1398] T dccce42f6cd64934808d8c6e5f366e35 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:53.923243 15474 ts_tablet_manager.cc:1398] T f8d3d130f4a44904924f7ccf4316fc75 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:53.923425 15474 tablet_bootstrap.cc:492] T dccce42f6cd64934808d8c6e5f366e35 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:53.924422 15474 tablet_bootstrap.cc:654] T dccce42f6cd64934808d8c6e5f366e35 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.935415 7256 tablet_service.cc:1467] Processing CreateTablet for tablet 26bdd19855f4422ca9af1ab92230af3d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c707d1c2dcd540e5849a49c633435cce]), partition=
I20240503 01:59:53.935581 9341 tablet_service.cc:1467] Processing CreateTablet for tablet 26bdd19855f4422ca9af1ab92230af3d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c707d1c2dcd540e5849a49c633435cce]), partition=
I20240503 01:59:53.935770 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26bdd19855f4422ca9af1ab92230af3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.935927 9341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26bdd19855f4422ca9af1ab92230af3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.936579 9735 tablet_service.cc:1467] Processing CreateTablet for tablet 26bdd19855f4422ca9af1ab92230af3d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c707d1c2dcd540e5849a49c633435cce]), partition=
I20240503 01:59:53.936817 9735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26bdd19855f4422ca9af1ab92230af3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:53.942857 15479 tablet_bootstrap.cc:492] T f8d3d130f4a44904924f7ccf4316fc75 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:53.942989 15479 ts_tablet_manager.cc:1367] T f8d3d130f4a44904924f7ccf4316fc75 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.072s user 0.002s sys 0.000s
I20240503 01:59:53.943506 15479 raft_consensus.cc:348] T f8d3d130f4a44904924f7ccf4316fc75 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:53.943641 15479 raft_consensus.cc:374] T f8d3d130f4a44904924f7ccf4316fc75 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.943672 15479 raft_consensus.cc:729] T f8d3d130f4a44904924f7ccf4316fc75 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.943791 15479 consensus_queue.cc:260] T f8d3d130f4a44904924f7ccf4316fc75 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:53.944131 15479 ts_tablet_manager.cc:1398] T f8d3d130f4a44904924f7ccf4316fc75 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:53.944470 15479 tablet_bootstrap.cc:492] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:53.946066 15479 tablet_bootstrap.cc:654] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.949221 15475 tablet_bootstrap.cc:492] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:53.949327 15475 ts_tablet_manager.cc:1367] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.064s user 0.002s sys 0.000s
I20240503 01:59:53.949771 15475 raft_consensus.cc:348] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:53.949901 15475 raft_consensus.cc:374] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.949927 15475 raft_consensus.cc:729] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.950044 15475 consensus_queue.cc:260] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:53.950433 15488 tablet_bootstrap.cc:492] T 26bdd19855f4422ca9af1ab92230af3d P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:53.952025 15475 ts_tablet_manager.cc:1398] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:53.952983 15476 tablet_bootstrap.cc:492] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:53.954965 15474 tablet_bootstrap.cc:492] T dccce42f6cd64934808d8c6e5f366e35 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:53.955070 15474 ts_tablet_manager.cc:1367] T dccce42f6cd64934808d8c6e5f366e35 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.000s
I20240503 01:59:53.955555 15474 raft_consensus.cc:348] T dccce42f6cd64934808d8c6e5f366e35 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:53.955670 15474 raft_consensus.cc:374] T dccce42f6cd64934808d8c6e5f366e35 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.955704 15474 raft_consensus.cc:729] T dccce42f6cd64934808d8c6e5f366e35 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.955832 15474 consensus_queue.cc:260] T dccce42f6cd64934808d8c6e5f366e35 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:53.956130 15474 ts_tablet_manager.cc:1398] T dccce42f6cd64934808d8c6e5f366e35 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:53.958571 15474 tablet_bootstrap.cc:492] T 26bdd19855f4422ca9af1ab92230af3d P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:53.959646 15474 tablet_bootstrap.cc:654] T 26bdd19855f4422ca9af1ab92230af3d P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.954165 15476 ts_tablet_manager.cc:1367] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.074s user 0.001s sys 0.001s
I20240503 01:59:53.962190 15488 tablet_bootstrap.cc:654] T 26bdd19855f4422ca9af1ab92230af3d P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:53.962265 15476 raft_consensus.cc:348] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:53.962409 15476 raft_consensus.cc:374] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.962447 15476 raft_consensus.cc:729] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.962644 15476 consensus_queue.cc:260] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:53.965121 15476 ts_tablet_manager.cc:1398] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 01:59:53.965359 15474 tablet_bootstrap.cc:492] T 26bdd19855f4422ca9af1ab92230af3d P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:53.965459 15474 ts_tablet_manager.cc:1367] T 26bdd19855f4422ca9af1ab92230af3d P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 01:59:53.965891 15474 raft_consensus.cc:348] T 26bdd19855f4422ca9af1ab92230af3d P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:53.966185 15474 raft_consensus.cc:374] T 26bdd19855f4422ca9af1ab92230af3d P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.966262 15474 raft_consensus.cc:729] T 26bdd19855f4422ca9af1ab92230af3d P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.966316 15479 tablet_bootstrap.cc:492] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:53.966435 15479 ts_tablet_manager.cc:1367] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240503 01:59:53.966496 15474 consensus_queue.cc:260] T 26bdd19855f4422ca9af1ab92230af3d P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:53.966890 15474 ts_tablet_manager.cc:1398] T 26bdd19855f4422ca9af1ab92230af3d P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:53.966980 15479 raft_consensus.cc:348] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:53.967283 15479 raft_consensus.cc:374] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.967355 15479 raft_consensus.cc:729] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.967514 15479 consensus_queue.cc:260] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:53.968731 15479 ts_tablet_manager.cc:1398] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:53.969017 9494 raft_consensus.cc:1203] T 96dd9b1f85cf471495dff16d5fa9148a P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240503 01:59:53.969894 7192 tablet.cc:2369] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:53.979000 9362 raft_consensus.cc:1203] T dcc773c6d5794e26a0003c7d4acc5d21 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:53.979171 15488 tablet_bootstrap.cc:492] T 26bdd19855f4422ca9af1ab92230af3d P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:53.979290 15488 ts_tablet_manager.cc:1367] T 26bdd19855f4422ca9af1ab92230af3d P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20240503 01:59:53.979852 15488 raft_consensus.cc:348] T 26bdd19855f4422ca9af1ab92230af3d P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:53.979997 15488 raft_consensus.cc:374] T 26bdd19855f4422ca9af1ab92230af3d P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:53.980044 15488 raft_consensus.cc:729] T 26bdd19855f4422ca9af1ab92230af3d P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:53.980204 15488 consensus_queue.cc:260] T 26bdd19855f4422ca9af1ab92230af3d P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:53.981344 15488 ts_tablet_manager.cc:1398] T 26bdd19855f4422ca9af1ab92230af3d P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:53.984956 14781 raft_consensus.cc:482] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:53.985095 14781 raft_consensus.cc:504] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:53.985622 14781 leader_election.cc:288] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:53.985947 9362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8d3d130f4a44904924f7ccf4316fc75" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:53.986037 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8d3d130f4a44904924f7ccf4316fc75" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:53.986093 9362 raft_consensus.cc:2454] T f8d3d130f4a44904924f7ccf4316fc75 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:53.986167 9755 raft_consensus.cc:2454] T f8d3d130f4a44904924f7ccf4316fc75 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:53.986462 6817 leader_election.cc:302] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:53.986668 14781 raft_consensus.cc:2781] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:53.986725 14781 raft_consensus.cc:482] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:53.986752 14781 raft_consensus.cc:3037] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.987942 14781 raft_consensus.cc:504] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:53.988314 14781 leader_election.cc:288] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Requested vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:53.988786 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8d3d130f4a44904924f7ccf4316fc75" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:53.988931 9755 raft_consensus.cc:3037] T f8d3d130f4a44904924f7ccf4316fc75 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.989293 9362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8d3d130f4a44904924f7ccf4316fc75" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:53.989425 9362 raft_consensus.cc:3037] T f8d3d130f4a44904924f7ccf4316fc75 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:53.990186 9362 raft_consensus.cc:2454] T f8d3d130f4a44904924f7ccf4316fc75 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:53.990482 9755 raft_consensus.cc:2454] T f8d3d130f4a44904924f7ccf4316fc75 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:53.990842 6818 leader_election.cc:302] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:53.991037 14781 raft_consensus.cc:2781] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:53.991392 14781 raft_consensus.cc:686] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 01:59:53.991544 14781 consensus_queue.cc:237] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:53.993901 6740 catalog_manager.cc:5484] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
W20240503 01:59:53.996465 7061 tablet.cc:2369] T d80398c9a264430197262bb42241d58d P 63331f455be049c1bdd6c23c59ac1bef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:54.000577 9494 raft_consensus.cc:1203] T 822bafcbc6de406997955ff7a1d9b612 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:54.050215 13349 raft_consensus.cc:482] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:54.050347 13349 raft_consensus.cc:504] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:54.050922 13349 leader_election.cc:288] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:54.051822 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26bdd19855f4422ca9af1ab92230af3d" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:54.051986 7275 raft_consensus.cc:2454] T 26bdd19855f4422ca9af1ab92230af3d P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:54.052253 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26bdd19855f4422ca9af1ab92230af3d" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:54.052368 9755 raft_consensus.cc:2454] T 26bdd19855f4422ca9af1ab92230af3d P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:54.053050 9297 leader_election.cc:302] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:54.053380 13349 raft_consensus.cc:2781] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:54.053442 13349 raft_consensus.cc:482] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:54.053483 13349 raft_consensus.cc:3037] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:54.054661 13349 raft_consensus.cc:504] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:54.055027 13349 leader_election.cc:288] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:54.056150 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26bdd19855f4422ca9af1ab92230af3d" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:54.056289 7275 raft_consensus.cc:3037] T 26bdd19855f4422ca9af1ab92230af3d P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:54.056557 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26bdd19855f4422ca9af1ab92230af3d" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:54.056768 9755 raft_consensus.cc:3037] T 26bdd19855f4422ca9af1ab92230af3d P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:54.057500 7275 raft_consensus.cc:2454] T 26bdd19855f4422ca9af1ab92230af3d P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:54.057936 9297 leader_election.cc:302] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 1d0e738c34af40fa9950c6f086619b3a; no voters:
I20240503 01:59:54.058193 9755 raft_consensus.cc:2454] T 26bdd19855f4422ca9af1ab92230af3d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:54.059540 15247 raft_consensus.cc:2781] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:54.059770 15247 raft_consensus.cc:686] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Running, Role: LEADER
I20240503 01:59:54.059973 15247 consensus_queue.cc:237] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:54.064283 9624 raft_consensus.cc:1203] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:54.064425 6734 catalog_manager.cc:5484] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
W20240503 01:59:54.074141 6930 tablet.cc:2369] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:54.105192 7143 raft_consensus.cc:1203] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 01:59:54.106410 15373 raft_consensus.cc:482] T dccce42f6cd64934808d8c6e5f366e35 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:54.106563 15373 raft_consensus.cc:504] T dccce42f6cd64934808d8c6e5f366e35 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:54.107057 15373 leader_election.cc:288] T dccce42f6cd64934808d8c6e5f366e35 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:54.107506 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dccce42f6cd64934808d8c6e5f366e35" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:54.107656 6881 raft_consensus.cc:2454] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 01:59:54.107966 9562 leader_election.cc:302] T dccce42f6cd64934808d8c6e5f366e35 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:54.108992 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dccce42f6cd64934808d8c6e5f366e35" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:54.109130 9755 raft_consensus.cc:2454] T dccce42f6cd64934808d8c6e5f366e35 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 01:59:54.109566 15373 raft_consensus.cc:2781] T dccce42f6cd64934808d8c6e5f366e35 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:54.109620 15373 raft_consensus.cc:482] T dccce42f6cd64934808d8c6e5f366e35 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:54.109653 15373 raft_consensus.cc:3037] T dccce42f6cd64934808d8c6e5f366e35 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:54.110888 15373 raft_consensus.cc:504] T dccce42f6cd64934808d8c6e5f366e35 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:54.111248 15373 leader_election.cc:288] T dccce42f6cd64934808d8c6e5f366e35 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Requested vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:54.111939 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dccce42f6cd64934808d8c6e5f366e35" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:54.112067 6881 raft_consensus.cc:3037] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:54.113734 6881 raft_consensus.cc:2454] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 01:59:54.114317 9562 leader_election.cc:302] T dccce42f6cd64934808d8c6e5f366e35 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:54.118340 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dccce42f6cd64934808d8c6e5f366e35" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:54.118609 9755 raft_consensus.cc:3037] T dccce42f6cd64934808d8c6e5f366e35 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:54.120023 15159 raft_consensus.cc:482] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:54.120141 15159 raft_consensus.cc:504] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:54.120191 9755 raft_consensus.cc:2454] T dccce42f6cd64934808d8c6e5f366e35 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 01:59:54.121014 15159 leader_election.cc:288] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:54.121323 14749 raft_consensus.cc:2781] T dccce42f6cd64934808d8c6e5f366e35 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:54.121500 14749 raft_consensus.cc:686] T dccce42f6cd64934808d8c6e5f366e35 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 LEADER]: Becoming Leader. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Running, Role: LEADER
I20240503 01:59:54.121454 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80398c9a264430197262bb42241d58d" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:54.121604 7012 raft_consensus.cc:2454] T d80398c9a264430197262bb42241d58d P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:54.121670 14749 consensus_queue.cc:237] T dccce42f6cd64934808d8c6e5f366e35 P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:54.121918 7078 leader_election.cc:302] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:54.122318 15163 raft_consensus.cc:2781] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:54.122366 15163 raft_consensus.cc:482] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:54.122400 15163 raft_consensus.cc:3037] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:54.123853 15163 raft_consensus.cc:504] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:54.124261 15163 leader_election.cc:288] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:54.121676 9624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80398c9a264430197262bb42241d58d" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:54.124464 9624 raft_consensus.cc:2454] T d80398c9a264430197262bb42241d58d P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:54.124552 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80398c9a264430197262bb42241d58d" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:54.124635 9625 raft_consensus.cc:3037] T d80398c9a264430197262bb42241d58d P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:54.124764 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80398c9a264430197262bb42241d58d" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:54.124889 7014 raft_consensus.cc:3037] T d80398c9a264430197262bb42241d58d P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:54.125766 9625 raft_consensus.cc:2454] T d80398c9a264430197262bb42241d58d P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:54.125805 7014 raft_consensus.cc:2454] T d80398c9a264430197262bb42241d58d P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:54.126230 7078 leader_election.cc:302] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:54.126464 15159 raft_consensus.cc:2781] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:54.126543 15159 raft_consensus.cc:686] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Becoming Leader. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Running, Role: LEADER
I20240503 01:59:54.126673 15159 consensus_queue.cc:237] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:54.128893 6732 catalog_manager.cc:5484] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:54.135147 6740 catalog_manager.cc:5484] T dccce42f6cd64934808d8c6e5f366e35 P b7062fbbc2b54fff9e99ec4ea5562c76 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:54.169621 9341 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 01:59:54.173538 7323 tablet.cc:2369] T 26bdd19855f4422ca9af1ab92230af3d P 1d0e738c34af40fa9950c6f086619b3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 01:59:54.180490 9473 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:54.184969 6740 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:54.186336 7256 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:54.192916 7275 raft_consensus.cc:1261] T 26bdd19855f4422ca9af1ab92230af3d P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:54.193460 15535 consensus_queue.cc:1035] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:54.193598 9755 raft_consensus.cc:1261] T 26bdd19855f4422ca9af1ab92230af3d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:54.193686 6993 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:54.194067 13349 consensus_queue.cc:1035] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:54.194190 9604 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:54.198163 6861 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:54.199693 6734 catalog_manager.cc:5576] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271) does not have the latest schema for tablet 26bdd19855f4422ca9af1ab92230af3d (table rebalancer_dynamic_table [id=c707d1c2dcd540e5849a49c633435cce]). Expected version 1 got 0
I20240503 01:59:54.201774 6733 catalog_manager.cc:5576] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053) does not have the latest schema for tablet 26bdd19855f4422ca9af1ab92230af3d (table rebalancer_dynamic_table [id=c707d1c2dcd540e5849a49c633435cce]). Expected version 1 got 0
I20240503 01:59:54.205927 7124 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:54.205927 6733 catalog_manager.cc:5576] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271) does not have the latest schema for tablet 26bdd19855f4422ca9af1ab92230af3d (table rebalancer_dynamic_table [id=c707d1c2dcd540e5849a49c633435cce]). Expected version 1 got 0
I20240503 01:59:54.214710 9735 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:54.215929 6732 catalog_manager.cc:5576] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053) does not have the latest schema for tablet 26bdd19855f4422ca9af1ab92230af3d (table rebalancer_dynamic_table [id=c707d1c2dcd540e5849a49c633435cce]). Expected version 1 got 0
I20240503 01:59:54.218221 15556 tablet.cc:1722] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:54.218611 15556 tablet.cc:1620] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 01:59:54.220813 15558 tablet.cc:1722] T 26bdd19855f4422ca9af1ab92230af3d P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:54.221149 15558 tablet.cc:1620] T 26bdd19855f4422ca9af1ab92230af3d P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:54.222036 15557 tablet.cc:1722] T 26bdd19855f4422ca9af1ab92230af3d P 1d0e738c34af40fa9950c6f086619b3a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:54.222379 15557 tablet.cc:1620] T 26bdd19855f4422ca9af1ab92230af3d P 1d0e738c34af40fa9950c6f086619b3a: MemRowSet was empty: no flush needed.
I20240503 01:59:54.232661 6733 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=c707d1c2dcd540e5849a49c633435cce] alter complete (version 1)
I20240503 01:59:54.255434 9494 raft_consensus.cc:1203] T 487eafe47df84b468618f473eca37874 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 01:59:54.301313 9362 consensus_queue.cc:237] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:54.302767 7143 raft_consensus.cc:1261] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:54.302767 9494 raft_consensus.cc:1261] T 23ae815010364a5586e60b77ff0a600a P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:54.303192 13349 consensus_queue.cc:1035] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:54.303409 15535 consensus_queue.cc:1035] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:54.306265 13349 raft_consensus.cc:2932] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:54.307719 7143 raft_consensus.cc:2932] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:54.311396 9494 raft_consensus.cc:2932] T 23ae815010364a5586e60b77ff0a600a P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:54.311672 6733 catalog_manager.cc:5484] T 23ae815010364a5586e60b77ff0a600a P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: config changed from index 2 to 3, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:54.315537 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet 23ae815010364a5586e60b77ff0a600a with delete_type TABLET_DATA_TOMBSTONED (TS 5dcc33cc621a4cbbae155f3230fb7c46 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:54.316159 15582 tablet_replica.cc:330] T 23ae815010364a5586e60b77ff0a600a P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:54.316331 15582 raft_consensus.cc:2229] T 23ae815010364a5586e60b77ff0a600a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:54.316507 15582 raft_consensus.cc:2258] T 23ae815010364a5586e60b77ff0a600a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:54.317067 15582 ts_tablet_manager.cc:1875] T 23ae815010364a5586e60b77ff0a600a P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:54.319835 15582 ts_tablet_manager.cc:1888] T 23ae815010364a5586e60b77ff0a600a P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 01:59:54.319916 15582 log.cc:1196] T 23ae815010364a5586e60b77ff0a600a P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/23ae815010364a5586e60b77ff0a600a
I20240503 01:59:54.320482 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 23ae815010364a5586e60b77ff0a600a (table rebalancer_extra_table_6 [id=73d68a2fa3594bc4b64c9e73724c9ae0]) successfully deleted
I20240503 01:59:54.335331 6732 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:54.349437 15556 tablet.cc:1722] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:54.349843 15556 tablet.cc:1620] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 01:59:54.350793 15557 tablet.cc:1722] T 26bdd19855f4422ca9af1ab92230af3d P 1d0e738c34af40fa9950c6f086619b3a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:54.351286 15557 tablet.cc:1620] T 26bdd19855f4422ca9af1ab92230af3d P 1d0e738c34af40fa9950c6f086619b3a: MemRowSet was empty: no flush needed.
I20240503 01:59:54.351739 15558 tablet.cc:1722] T 26bdd19855f4422ca9af1ab92230af3d P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:54.352128 15558 tablet.cc:1620] T 26bdd19855f4422ca9af1ab92230af3d P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:54.359655 6732 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=c707d1c2dcd540e5849a49c633435cce] alter complete (version 2)
I20240503 01:59:54.362473 9494 raft_consensus.cc:1203] T fb287629baab4cada193ae57695ef1b5 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 01:59:54.374015 14781 consensus_queue.cc:1035] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:54.377810 14781 consensus_queue.cc:1035] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:54.458639 9341 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 01:59:54.462992 9604 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:54.463006 6993 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:54.474757 6736 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:54.479473 9735 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:54.485028 15556 tablet.cc:1722] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:54.485532 15558 tablet.cc:1722] T 26bdd19855f4422ca9af1ab92230af3d P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:54.485890 15557 tablet.cc:1722] T 26bdd19855f4422ca9af1ab92230af3d P 1d0e738c34af40fa9950c6f086619b3a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:54.500727 7124 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:54.506134 7256 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:54.506309 15163 consensus_queue.cc:1035] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:54.505192 6736 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=c707d1c2dcd540e5849a49c633435cce] alter complete (version 3)
I20240503 01:59:54.508301 9473 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:54.527944 15159 consensus_queue.cc:1035] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:54.546178 6861 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:54.596271 7012 consensus_queue.cc:237] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:54.597653 6882 raft_consensus.cc:1261] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:54.597707 9624 raft_consensus.cc:1261] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:54.598235 15227 consensus_queue.cc:1035] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:54.598644 15589 consensus_queue.cc:1035] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:54.601135 14749 consensus_queue.cc:1035] T dccce42f6cd64934808d8c6e5f366e35 P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:54.601228 15589 raft_consensus.cc:2932] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:54.602337 9624 raft_consensus.cc:2932] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:54.602389 6882 raft_consensus.cc:2932] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:54.604615 6736 catalog_manager.cc:5484] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:54.609066 7124 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c3d43c8b2ca42b19616fde1d7eb47a3 with delete_type TABLET_DATA_TOMBSTONED (TS c688f3df56ac4b6d9b0c30fa4cc8ca2a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:54.609395 6736 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:54.609550 6736 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:54.609884 15658 tablet_replica.cc:330] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:54.610040 15658 raft_consensus.cc:2229] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:54.610180 15658 raft_consensus.cc:2258] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:54.610905 15658 ts_tablet_manager.cc:1875] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:54.612910 6736 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 26bdd19855f4422ca9af1ab92230af3d
I20240503 01:59:54.613623 15658 ts_tablet_manager.cc:1888] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 01:59:54.613766 15658 log.cc:1196] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/8c3d43c8b2ca42b19616fde1d7eb47a3
I20240503 01:59:54.614085 7256 tablet_service.cc:1514] Processing DeleteTablet for tablet 26bdd19855f4422ca9af1ab92230af3d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:54 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:54.614238 9341 tablet_service.cc:1514] Processing DeleteTablet for tablet 26bdd19855f4422ca9af1ab92230af3d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:54 UTC) from {username='slave'} at 127.0.0.1:36040
I20240503 01:59:54.614470 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet 8c3d43c8b2ca42b19616fde1d7eb47a3 (table rebalance_test_table_1 [id=38e214942900477190e2c3b67ccc5952]) successfully deleted
I20240503 01:59:54.614605 9735 tablet_service.cc:1514] Processing DeleteTablet for tablet 26bdd19855f4422ca9af1ab92230af3d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:54 UTC) from {username='slave'} at 127.0.0.1:48152
I20240503 01:59:54.614912 15665 tablet_replica.cc:330] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8: stopping tablet replica
I20240503 01:59:54.614912 15666 tablet_replica.cc:330] T 26bdd19855f4422ca9af1ab92230af3d P db1a0000aaa346f79c0aa4b6c5105191: stopping tablet replica
I20240503 01:59:54.615069 15666 raft_consensus.cc:2229] T 26bdd19855f4422ca9af1ab92230af3d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:54.615105 15665 raft_consensus.cc:2229] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:54.615293 15666 raft_consensus.cc:2258] T 26bdd19855f4422ca9af1ab92230af3d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:54.615393 15665 raft_consensus.cc:2258] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:54.616398 15665 ts_tablet_manager.cc:1875] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:54.616722 15375 consensus_queue.cc:1035] T dccce42f6cd64934808d8c6e5f366e35 P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:54.616808 15666 ts_tablet_manager.cc:1875] T 26bdd19855f4422ca9af1ab92230af3d P db1a0000aaa346f79c0aa4b6c5105191: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:54.618757 15664 tablet_replica.cc:330] T 26bdd19855f4422ca9af1ab92230af3d P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:54.618973 15664 raft_consensus.cc:2229] T 26bdd19855f4422ca9af1ab92230af3d P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:54.619163 15664 raft_consensus.cc:2258] T 26bdd19855f4422ca9af1ab92230af3d P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:54.619185 15665 ts_tablet_manager.cc:1888] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:54.619253 15665 log.cc:1196] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/26bdd19855f4422ca9af1ab92230af3d
I20240503 01:59:54.619626 15665 ts_tablet_manager.cc:1909] T 26bdd19855f4422ca9af1ab92230af3d P 12e8c683db054df3b75c6285b98a5ad8: Deleting consensus metadata
I20240503 01:59:54.621215 6727 catalog_manager.cc:4830] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): tablet 26bdd19855f4422ca9af1ab92230af3d (table rebalancer_dynamic_table_new_name [id=c707d1c2dcd540e5849a49c633435cce]) successfully deleted
I20240503 01:59:54.621798 15664 ts_tablet_manager.cc:1875] T 26bdd19855f4422ca9af1ab92230af3d P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:54.622635 15666 ts_tablet_manager.cc:1888] T 26bdd19855f4422ca9af1ab92230af3d P db1a0000aaa346f79c0aa4b6c5105191: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:54.622778 15666 log.cc:1196] T 26bdd19855f4422ca9af1ab92230af3d P db1a0000aaa346f79c0aa4b6c5105191: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/26bdd19855f4422ca9af1ab92230af3d
I20240503 01:59:54.623126 15666 ts_tablet_manager.cc:1909] T 26bdd19855f4422ca9af1ab92230af3d P db1a0000aaa346f79c0aa4b6c5105191: Deleting consensus metadata
I20240503 01:59:54.624670 6726 catalog_manager.cc:4830] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): tablet 26bdd19855f4422ca9af1ab92230af3d (table rebalancer_dynamic_table_new_name [id=c707d1c2dcd540e5849a49c633435cce]) successfully deleted
I20240503 01:59:54.626029 15664 ts_tablet_manager.cc:1888] T 26bdd19855f4422ca9af1ab92230af3d P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:54.626113 15664 log.cc:1196] T 26bdd19855f4422ca9af1ab92230af3d P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/26bdd19855f4422ca9af1ab92230af3d
I20240503 01:59:54.626468 15664 ts_tablet_manager.cc:1909] T 26bdd19855f4422ca9af1ab92230af3d P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:54.627264 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet 26bdd19855f4422ca9af1ab92230af3d (table rebalancer_dynamic_table_new_name [id=c707d1c2dcd540e5849a49c633435cce]) successfully deleted
I20240503 01:59:54.709554 6741 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:54.714818 6737 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:54.718127 7124 tablet_service.cc:1467] Processing CreateTablet for tablet 8db6c1313c0042d48416b4f38fe7166f (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=b15f374995474d32bb2e755f0bcb94a5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:54.718411 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8db6c1313c0042d48416b4f38fe7166f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.720918 9604 tablet_service.cc:1467] Processing CreateTablet for tablet 8db6c1313c0042d48416b4f38fe7166f (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=b15f374995474d32bb2e755f0bcb94a5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:54.720927 9473 tablet_service.cc:1467] Processing CreateTablet for tablet 9b8976486acf49d5a9d0d496379ee2a6 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=b15f374995474d32bb2e755f0bcb94a5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:54.721225 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b8976486acf49d5a9d0d496379ee2a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.721237 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8db6c1313c0042d48416b4f38fe7166f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.718075 7125 tablet_service.cc:1467] Processing CreateTablet for tablet 516415bcf2ba44bbb6356731427ee2ec (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=b15f374995474d32bb2e755f0bcb94a5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:54.721567 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 516415bcf2ba44bbb6356731427ee2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.721596 9606 tablet_service.cc:1467] Processing CreateTablet for tablet 516415bcf2ba44bbb6356731427ee2ec (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=b15f374995474d32bb2e755f0bcb94a5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:54.721789 9606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 516415bcf2ba44bbb6356731427ee2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.723783 9605 tablet_service.cc:1467] Processing CreateTablet for tablet 9b8976486acf49d5a9d0d496379ee2a6 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=b15f374995474d32bb2e755f0bcb94a5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:54.723999 9605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b8976486acf49d5a9d0d496379ee2a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.726269 9341 tablet_service.cc:1467] Processing CreateTablet for tablet 9b8976486acf49d5a9d0d496379ee2a6 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=b15f374995474d32bb2e755f0bcb94a5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:54.726799 9341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b8976486acf49d5a9d0d496379ee2a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.729099 15708 tablet_bootstrap.cc:492] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:54.729357 15705 tablet_bootstrap.cc:492] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:54.729704 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 4b11ba442cf74fccb4635dd019c0c8d8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=72681f43e34d44a484c1b8ddfd8fb6e5]), partition=
I20240503 01:59:54.729959 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b11ba442cf74fccb4635dd019c0c8d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.730276 7125 tablet_service.cc:1467] Processing CreateTablet for tablet 4b11ba442cf74fccb4635dd019c0c8d8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=72681f43e34d44a484c1b8ddfd8fb6e5]), partition=
I20240503 01:59:54.730506 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b11ba442cf74fccb4635dd019c0c8d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.730790 6993 tablet_service.cc:1467] Processing CreateTablet for tablet 4b11ba442cf74fccb4635dd019c0c8d8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=72681f43e34d44a484c1b8ddfd8fb6e5]), partition=
I20240503 01:59:54.731019 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b11ba442cf74fccb4635dd019c0c8d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.732146 15707 tablet_bootstrap.cc:492] T 9b8976486acf49d5a9d0d496379ee2a6 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:54.732959 15708 tablet_bootstrap.cc:654] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:54.733527 15707 tablet_bootstrap.cc:654] T 9b8976486acf49d5a9d0d496379ee2a6 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:54.733852 15717 tablet_bootstrap.cc:492] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:54.734131 15705 tablet_bootstrap.cc:654] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:54.735106 15717 tablet_bootstrap.cc:654] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:54.735685 15707 tablet_bootstrap.cc:492] T 9b8976486acf49d5a9d0d496379ee2a6 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:54.735841 15707 ts_tablet_manager.cc:1367] T 9b8976486acf49d5a9d0d496379ee2a6 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:54.736373 15707 raft_consensus.cc:348] T 9b8976486acf49d5a9d0d496379ee2a6 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:54.736622 15707 raft_consensus.cc:374] T 9b8976486acf49d5a9d0d496379ee2a6 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:54.736711 15707 raft_consensus.cc:729] T 9b8976486acf49d5a9d0d496379ee2a6 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:54.736871 15707 consensus_queue.cc:260] T 9b8976486acf49d5a9d0d496379ee2a6 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:54.737572 15707 ts_tablet_manager.cc:1398] T 9b8976486acf49d5a9d0d496379ee2a6 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:54.738003 9735 tablet_service.cc:1467] Processing CreateTablet for tablet 516415bcf2ba44bbb6356731427ee2ec (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=b15f374995474d32bb2e755f0bcb94a5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:54.738005 7256 tablet_service.cc:1467] Processing CreateTablet for tablet 8db6c1313c0042d48416b4f38fe7166f (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=b15f374995474d32bb2e755f0bcb94a5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:54.738301 9735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 516415bcf2ba44bbb6356731427ee2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.738299 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8db6c1313c0042d48416b4f38fe7166f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:54.738574 15717 tablet_bootstrap.cc:492] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:54.738672 15717 ts_tablet_manager.cc:1367] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:54.739105 15717 raft_consensus.cc:348] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:54.739205 15717 raft_consensus.cc:374] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:54.739234 15717 raft_consensus.cc:729] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:54.739385 15717 consensus_queue.cc:260] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:54.740125 15717 ts_tablet_manager.cc:1398] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:54.740725 15723 tablet_bootstrap.cc:492] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:54.742137 15723 tablet_bootstrap.cc:654] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:54.743428 15725 tablet_bootstrap.cc:492] T 8db6c1313c0042d48416b4f38fe7166f P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:54.743458 15706 tablet_bootstrap.cc:492] T 8db6c1313c0042d48416b4f38fe7166f P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:54.744390 15706 tablet_bootstrap.cc:654] T 8db6c1313c0042d48416b4f38fe7166f P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:54.744721 15725 tablet_bootstrap.cc:654] T 8db6c1313c0042d48416b4f38fe7166f P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:54.745211 15718 tablet_bootstrap.cc:492] T 4b11ba442cf74fccb4635dd019c0c8d8 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:54.746171 15718 tablet_bootstrap.cc:654] T 4b11ba442cf74fccb4635dd019c0c8d8 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:54.747386 15718 tablet_bootstrap.cc:492] T 4b11ba442cf74fccb4635dd019c0c8d8 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:54.747480 15718 ts_tablet_manager.cc:1367] T 4b11ba442cf74fccb4635dd019c0c8d8 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:54.747629 15705 tablet_bootstrap.cc:492] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:54.747658 15708 tablet_bootstrap.cc:492] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:54.747742 15705 ts_tablet_manager.cc:1367] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240503 01:59:54.747761 15708 ts_tablet_manager.cc:1367] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 01:59:54.748023 15718 raft_consensus.cc:348] T 4b11ba442cf74fccb4635dd019c0c8d8 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:54.748136 15718 raft_consensus.cc:374] T 4b11ba442cf74fccb4635dd019c0c8d8 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:54.748169 15718 raft_consensus.cc:729] T 4b11ba442cf74fccb4635dd019c0c8d8 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:54.748191 15708 raft_consensus.cc:348] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:54.748281 15708 raft_consensus.cc:374] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:54.748314 15708 raft_consensus.cc:729] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:54.748294 15718 consensus_queue.cc:260] T 4b11ba442cf74fccb4635dd019c0c8d8 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:54.748304 15705 raft_consensus.cc:348] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:54.748414 15705 raft_consensus.cc:374] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:54.748451 15705 raft_consensus.cc:729] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:54.748427 15708 consensus_queue.cc:260] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:54.748597 15718 ts_tablet_manager.cc:1398] T 4b11ba442cf74fccb4635dd019c0c8d8 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:54.748577 15705 consensus_queue.cc:260] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:54.748700 15708 ts_tablet_manager.cc:1398] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:54.748867 15705 ts_tablet_manager.cc:1398] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:54.749065 15705 tablet_bootstrap.cc:492] T 516415bcf2ba44bbb6356731427ee2ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:54.749796 15723 tablet_bootstrap.cc:492] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:54.749881 15723 ts_tablet_manager.cc:1367] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20240503 01:59:54.750049 15705 tablet_bootstrap.cc:654] T 516415bcf2ba44bbb6356731427ee2ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:54.750334 15723 raft_consensus.cc:348] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:54.750438 15723 raft_consensus.cc:374] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:54.750468 15723 raft_consensus.cc:729] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:54.750618 15723 consensus_queue.cc:260] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:54.750996 15723 ts_tablet_manager.cc:1398] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:54.769973 15706 tablet_bootstrap.cc:492] T 8db6c1313c0042d48416b4f38fe7166f P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:54.770109 15706 ts_tablet_manager.cc:1367] T 8db6c1313c0042d48416b4f38fe7166f P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20240503 01:59:54.770690 15706 raft_consensus.cc:348] T 8db6c1313c0042d48416b4f38fe7166f P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:54.770815 15706 raft_consensus.cc:374] T 8db6c1313c0042d48416b4f38fe7166f P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:54.770848 15706 raft_consensus.cc:729] T 8db6c1313c0042d48416b4f38fe7166f P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:54.770988 15706 consensus_queue.cc:260] T 8db6c1313c0042d48416b4f38fe7166f P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:54.771286 15706 ts_tablet_manager.cc:1398] T 8db6c1313c0042d48416b4f38fe7166f P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:54.771420 15706 tablet_bootstrap.cc:492] T 516415bcf2ba44bbb6356731427ee2ec P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:54.772936 15706 tablet_bootstrap.cc:654] T 516415bcf2ba44bbb6356731427ee2ec P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:54.773039 15705 tablet_bootstrap.cc:492] T 516415bcf2ba44bbb6356731427ee2ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:54.773144 15705 ts_tablet_manager.cc:1367] T 516415bcf2ba44bbb6356731427ee2ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.001s
I20240503 01:59:54.773643 15705 raft_consensus.cc:348] T 516415bcf2ba44bbb6356731427ee2ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:54.773749 15705 raft_consensus.cc:374] T 516415bcf2ba44bbb6356731427ee2ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:54.773779 15705 raft_consensus.cc:729] T 516415bcf2ba44bbb6356731427ee2ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:54.773929 15705 consensus_queue.cc:260] T 516415bcf2ba44bbb6356731427ee2ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:54.774283 15705 ts_tablet_manager.cc:1398] T 516415bcf2ba44bbb6356731427ee2ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:54.774466 15705 tablet_bootstrap.cc:492] T 4b11ba442cf74fccb4635dd019c0c8d8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:54.775101 9473 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:54.778704 7256 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:54.781435 9341 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:54.784515 15706 tablet_bootstrap.cc:492] T 516415bcf2ba44bbb6356731427ee2ec P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:54.784615 15706 ts_tablet_manager.cc:1367] T 516415bcf2ba44bbb6356731427ee2ec P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 01:59:54.785082 15706 raft_consensus.cc:348] T 516415bcf2ba44bbb6356731427ee2ec P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:54.785192 15706 raft_consensus.cc:374] T 516415bcf2ba44bbb6356731427ee2ec P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:54.785220 15706 raft_consensus.cc:729] T 516415bcf2ba44bbb6356731427ee2ec P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:54.785318 15706 consensus_queue.cc:260] T 516415bcf2ba44bbb6356731427ee2ec P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:54.785574 15706 ts_tablet_manager.cc:1398] T 516415bcf2ba44bbb6356731427ee2ec P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:54.785751 15706 tablet_bootstrap.cc:492] T 9b8976486acf49d5a9d0d496379ee2a6 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:54.786724 15706 tablet_bootstrap.cc:654] T 9b8976486acf49d5a9d0d496379ee2a6 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:54.788650 15706 tablet_bootstrap.cc:492] T 9b8976486acf49d5a9d0d496379ee2a6 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:54.788748 15706 ts_tablet_manager.cc:1367] T 9b8976486acf49d5a9d0d496379ee2a6 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:54.789206 15706 raft_consensus.cc:348] T 9b8976486acf49d5a9d0d496379ee2a6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:54.789321 15706 raft_consensus.cc:374] T 9b8976486acf49d5a9d0d496379ee2a6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:54.789358 15706 raft_consensus.cc:729] T 9b8976486acf49d5a9d0d496379ee2a6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:54.791982 15706 consensus_queue.cc:260] T 9b8976486acf49d5a9d0d496379ee2a6 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:54.792867 15706 ts_tablet_manager.cc:1398] T 9b8976486acf49d5a9d0d496379ee2a6 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:54.798707 7125 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:54.800212 6993 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:54.801076 6861 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:54.801733 15383 raft_consensus.cc:482] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:54.801832 15383 raft_consensus.cc:504] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:54.802309 15383 leader_election.cc:288] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:54.803122 9624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516415bcf2ba44bbb6356731427ee2ec" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:54.803262 9624 raft_consensus.cc:2454] T 516415bcf2ba44bbb6356731427ee2ec P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 01:59:54.803527 9692 leader_election.cc:302] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:54.804072 15383 raft_consensus.cc:2781] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:54.804147 15383 raft_consensus.cc:482] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:54.804188 15383 raft_consensus.cc:3037] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:54.805650 15383 raft_consensus.cc:504] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:54.806025 15383 leader_election.cc:288] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Requested vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:54.806421 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516415bcf2ba44bbb6356731427ee2ec" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:54.806566 7143 raft_consensus.cc:3037] T 516415bcf2ba44bbb6356731427ee2ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:54.806548 9624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516415bcf2ba44bbb6356731427ee2ec" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:54.806670 9624 raft_consensus.cc:3037] T 516415bcf2ba44bbb6356731427ee2ec P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:54.807588 7143 raft_consensus.cc:2454] T 516415bcf2ba44bbb6356731427ee2ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:54.807785 9624 raft_consensus.cc:2454] T 516415bcf2ba44bbb6356731427ee2ec P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:54.807884 9692 leader_election.cc:302] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:54.810961 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516415bcf2ba44bbb6356731427ee2ec" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:54.811107 7145 raft_consensus.cc:2362] T 516415bcf2ba44bbb6356731427ee2ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1. Re-sending same reply.
I20240503 01:59:54.812216 15383 raft_consensus.cc:2781] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:54.813607 15725 tablet_bootstrap.cc:492] T 8db6c1313c0042d48416b4f38fe7166f P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:54.813714 15725 ts_tablet_manager.cc:1367] T 8db6c1313c0042d48416b4f38fe7166f P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.070s user 0.002s sys 0.000s
I20240503 01:59:54.812319 15383 raft_consensus.cc:686] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Becoming Leader. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Running, Role: LEADER
I20240503 01:59:54.813917 15383 consensus_queue.cc:237] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:54.814177 15725 raft_consensus.cc:348] T 8db6c1313c0042d48416b4f38fe7166f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:54.814282 15725 raft_consensus.cc:374] T 8db6c1313c0042d48416b4f38fe7166f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:54.814321 15725 raft_consensus.cc:729] T 8db6c1313c0042d48416b4f38fe7166f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:54.814451 15725 consensus_queue.cc:260] T 8db6c1313c0042d48416b4f38fe7166f P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:54.814752 15725 ts_tablet_manager.cc:1398] T 8db6c1313c0042d48416b4f38fe7166f P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:54.814798 15705 tablet_bootstrap.cc:654] T 4b11ba442cf74fccb4635dd019c0c8d8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:54.820713 6733 catalog_manager.cc:5484] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: term changed from 0 to 1, leader changed from <none> to db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:54.820885 9735 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:54.841447 15705 tablet_bootstrap.cc:492] T 4b11ba442cf74fccb4635dd019c0c8d8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:54.841588 15705 ts_tablet_manager.cc:1367] T 4b11ba442cf74fccb4635dd019c0c8d8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.067s user 0.002s sys 0.000s
I20240503 01:59:54.842151 15705 raft_consensus.cc:348] T 4b11ba442cf74fccb4635dd019c0c8d8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:54.843721 15705 raft_consensus.cc:374] T 4b11ba442cf74fccb4635dd019c0c8d8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:54.843793 15705 raft_consensus.cc:729] T 4b11ba442cf74fccb4635dd019c0c8d8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:54.844008 15705 consensus_queue.cc:260] T 4b11ba442cf74fccb4635dd019c0c8d8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:54.844362 9605 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:54.844403 15705 ts_tablet_manager.cc:1398] T 4b11ba442cf74fccb4635dd019c0c8d8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 01:59:54.857877 15373 raft_consensus.cc:482] T 8db6c1313c0042d48416b4f38fe7166f P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:54.858129 15373 raft_consensus.cc:504] T 8db6c1313c0042d48416b4f38fe7166f P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:54.858675 15373 leader_election.cc:288] T 8db6c1313c0042d48416b4f38fe7166f P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:54.859513 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db6c1313c0042d48416b4f38fe7166f" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:54.859684 7275 raft_consensus.cc:2454] T 8db6c1313c0042d48416b4f38fe7166f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 01:59:54.860008 9560 leader_election.cc:302] T 8db6c1313c0042d48416b4f38fe7166f P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:54.860190 15373 raft_consensus.cc:2781] T 8db6c1313c0042d48416b4f38fe7166f P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:54.860244 15373 raft_consensus.cc:482] T 8db6c1313c0042d48416b4f38fe7166f P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:54.860287 15373 raft_consensus.cc:3037] T 8db6c1313c0042d48416b4f38fe7166f P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:54.860364 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db6c1313c0042d48416b4f38fe7166f" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:54.860504 7143 raft_consensus.cc:2454] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 01:59:54.861567 15373 raft_consensus.cc:504] T 8db6c1313c0042d48416b4f38fe7166f P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:54.862057 15668 raft_consensus.cc:482] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:54.862160 15668 raft_consensus.cc:504] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:54.862543 15668 leader_election.cc:288] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:54.862567 15373 leader_election.cc:288] T 8db6c1313c0042d48416b4f38fe7166f P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:54.862968 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db6c1313c0042d48416b4f38fe7166f" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:54.863098 7275 raft_consensus.cc:3037] T 8db6c1313c0042d48416b4f38fe7166f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:54.863152 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b11ba442cf74fccb4635dd019c0c8d8" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:54.863291 7012 raft_consensus.cc:2454] T 4b11ba442cf74fccb4635dd019c0c8d8 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:54.864192 7275 raft_consensus.cc:2454] T 8db6c1313c0042d48416b4f38fe7166f P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 01:59:54.864429 6816 leader_election.cc:302] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:54.864486 9560 leader_election.cc:302] T 8db6c1313c0042d48416b4f38fe7166f P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:54.864631 15373 raft_consensus.cc:2781] T 8db6c1313c0042d48416b4f38fe7166f P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:54.864735 15373 raft_consensus.cc:686] T 8db6c1313c0042d48416b4f38fe7166f P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 LEADER]: Becoming Leader. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Running, Role: LEADER
I20240503 01:59:54.864881 15373 consensus_queue.cc:237] T 8db6c1313c0042d48416b4f38fe7166f P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:54.865605 14781 raft_consensus.cc:2781] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:54.865670 14781 raft_consensus.cc:482] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:54.865713 14781 raft_consensus.cc:3037] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:54.865733 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db6c1313c0042d48416b4f38fe7166f" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:54.865777 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b11ba442cf74fccb4635dd019c0c8d8" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:54.865866 7143 raft_consensus.cc:3037] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:54.865876 7145 raft_consensus.cc:2454] T 4b11ba442cf74fccb4635dd019c0c8d8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:54.866832 14781 raft_consensus.cc:504] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:54.867178 14781 leader_election.cc:288] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Requested vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:54.867504 7143 raft_consensus.cc:2454] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 01:59:54.867594 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b11ba442cf74fccb4635dd019c0c8d8" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:54.867720 7145 raft_consensus.cc:3037] T 4b11ba442cf74fccb4635dd019c0c8d8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:54.868646 7145 raft_consensus.cc:2454] T 4b11ba442cf74fccb4635dd019c0c8d8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:54.868913 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b11ba442cf74fccb4635dd019c0c8d8" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:54.869055 7012 raft_consensus.cc:3037] T 4b11ba442cf74fccb4635dd019c0c8d8 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:54.870074 7012 raft_consensus.cc:2454] T 4b11ba442cf74fccb4635dd019c0c8d8 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:54.870371 6816 leader_election.cc:302] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:54.870605 14781 raft_consensus.cc:2781] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:54.870682 14781 raft_consensus.cc:686] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 01:59:54.870816 14781 consensus_queue.cc:237] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:54.871587 6733 catalog_manager.cc:5484] T 8db6c1313c0042d48416b4f38fe7166f P b7062fbbc2b54fff9e99ec4ea5562c76 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:54.873000 6737 catalog_manager.cc:5484] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:54.925719 7012 consensus_queue.cc:237] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:54.927207 9494 raft_consensus.cc:1261] T 96dd9b1f85cf471495dff16d5fa9148a P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:54.927454 7145 raft_consensus.cc:1261] T 96dd9b1f85cf471495dff16d5fa9148a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:54.927685 15227 consensus_queue.cc:1035] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:54.927915 14833 consensus_queue.cc:1035] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:54.929944 15589 raft_consensus.cc:2932] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:54.930408 9494 raft_consensus.cc:2932] T 96dd9b1f85cf471495dff16d5fa9148a P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:54.930852 7145 raft_consensus.cc:2932] T 96dd9b1f85cf471495dff16d5fa9148a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:54.934839 6733 catalog_manager.cc:5484] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index 4 to 5, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:54.938977 7256 tablet_service.cc:1514] Processing DeleteTablet for tablet 96dd9b1f85cf471495dff16d5fa9148a with delete_type TABLET_DATA_TOMBSTONED (TS 1d0e738c34af40fa9950c6f086619b3a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:54.939242 15664 tablet_replica.cc:330] T 96dd9b1f85cf471495dff16d5fa9148a P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:54.939393 15664 raft_consensus.cc:2229] T 96dd9b1f85cf471495dff16d5fa9148a P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:54.939553 15664 raft_consensus.cc:2258] T 96dd9b1f85cf471495dff16d5fa9148a P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:54.940181 15664 ts_tablet_manager.cc:1875] T 96dd9b1f85cf471495dff16d5fa9148a P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:54.942412 15664 ts_tablet_manager.cc:1888] T 96dd9b1f85cf471495dff16d5fa9148a P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 01:59:54.942502 15664 log.cc:1196] T 96dd9b1f85cf471495dff16d5fa9148a P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/96dd9b1f85cf471495dff16d5fa9148a
I20240503 01:59:54.943117 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet 96dd9b1f85cf471495dff16d5fa9148a (table rebalance_test_table_11 [id=dce2b86d058140cfb0a0d5088d32ec71]) successfully deleted
I20240503 01:59:54.995859 6733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:54.998178 13349 raft_consensus.cc:482] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:54.998325 13349 raft_consensus.cc:504] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:54.998853 13349 leader_election.cc:288] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:54.999737 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b8976486acf49d5a9d0d496379ee2a6" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:54.999887 9494 raft_consensus.cc:2454] T 9b8976486acf49d5a9d0d496379ee2a6 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:55.000443 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b8976486acf49d5a9d0d496379ee2a6" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:55.000586 9625 raft_consensus.cc:2454] T 9b8976486acf49d5a9d0d496379ee2a6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:55.000917 9298 leader_election.cc:302] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:55.001094 13349 raft_consensus.cc:2781] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:55.001154 13349 raft_consensus.cc:482] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:55.001196 13349 raft_consensus.cc:3037] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:55.002312 13349 raft_consensus.cc:504] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:55.002689 13349 leader_election.cc:288] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Requested vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:55.003320 7012 raft_consensus.cc:1261] T 4b11ba442cf74fccb4635dd019c0c8d8 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:55.003777 7145 raft_consensus.cc:1261] T 4b11ba442cf74fccb4635dd019c0c8d8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:55.004046 15668 consensus_queue.cc:1035] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:55.004324 14781 consensus_queue.cc:1035] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:55.004889 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b8976486acf49d5a9d0d496379ee2a6" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:55.005029 9625 raft_consensus.cc:3037] T 9b8976486acf49d5a9d0d496379ee2a6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:55.006330 9625 raft_consensus.cc:2454] T 9b8976486acf49d5a9d0d496379ee2a6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:55.006419 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b8976486acf49d5a9d0d496379ee2a6" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:55.006565 9494 raft_consensus.cc:3037] T 9b8976486acf49d5a9d0d496379ee2a6 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:55.006744 9298 leader_election.cc:302] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:55.006937 13349 raft_consensus.cc:2781] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:55.007005 13349 raft_consensus.cc:686] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Running, Role: LEADER
I20240503 01:59:55.007153 13349 consensus_queue.cc:237] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:55.007799 9494 raft_consensus.cc:2454] T 9b8976486acf49d5a9d0d496379ee2a6 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:55.010669 6733 catalog_manager.cc:5484] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:55.011024 15802 tablet.cc:1722] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:55.011523 15802 tablet.cc:1620] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46: MemRowSet was empty: no flush needed.
I20240503 01:59:55.012713 6741 catalog_manager.cc:5576] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231) does not have the latest schema for tablet 4b11ba442cf74fccb4635dd019c0c8d8 (table rebalancer_dynamic_table [id=72681f43e34d44a484c1b8ddfd8fb6e5]). Expected version 1 got 0
I20240503 01:59:55.016831 6732 catalog_manager.cc:5576] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979) does not have the latest schema for tablet 4b11ba442cf74fccb4635dd019c0c8d8 (table rebalancer_dynamic_table [id=72681f43e34d44a484c1b8ddfd8fb6e5]). Expected version 1 got 0
I20240503 01:59:55.018137 15804 tablet.cc:1722] T 4b11ba442cf74fccb4635dd019c0c8d8 P 63331f455be049c1bdd6c23c59ac1bef: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:55.019376 15804 tablet.cc:1620] T 4b11ba442cf74fccb4635dd019c0c8d8 P 63331f455be049c1bdd6c23c59ac1bef: MemRowSet was empty: no flush needed.
I20240503 01:59:55.019692 6741 catalog_manager.cc:5576] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979) does not have the latest schema for tablet 4b11ba442cf74fccb4635dd019c0c8d8 (table rebalancer_dynamic_table [id=72681f43e34d44a484c1b8ddfd8fb6e5]). Expected version 1 got 0
I20240503 01:59:55.020156 6739 catalog_manager.cc:5576] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231) does not have the latest schema for tablet 4b11ba442cf74fccb4635dd019c0c8d8 (table rebalancer_dynamic_table [id=72681f43e34d44a484c1b8ddfd8fb6e5]). Expected version 1 got 0
I20240503 01:59:55.023926 15803 tablet.cc:1722] T 4b11ba442cf74fccb4635dd019c0c8d8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:55.024298 15803 tablet.cc:1620] T 4b11ba442cf74fccb4635dd019c0c8d8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: MemRowSet was empty: no flush needed.
I20240503 01:59:55.030166 6734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=72681f43e34d44a484c1b8ddfd8fb6e5] alter complete (version 1)
I20240503 01:59:55.112902 6861 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:55.122731 9473 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:55.123749 6993 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:55.130298 9605 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:55.131014 7256 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:55.132469 6733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:55.138443 9735 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:55.146102 9341 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 01:59:55.149474 15802 tablet.cc:1722] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:55.149793 15803 tablet.cc:1722] T 4b11ba442cf74fccb4635dd019c0c8d8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:55.149873 15802 tablet.cc:1620] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46: MemRowSet was empty: no flush needed.
I20240503 01:59:55.150120 15803 tablet.cc:1620] T 4b11ba442cf74fccb4635dd019c0c8d8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: MemRowSet was empty: no flush needed.
I20240503 01:59:55.150964 15804 tablet.cc:1722] T 4b11ba442cf74fccb4635dd019c0c8d8 P 63331f455be049c1bdd6c23c59ac1bef: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:55.151304 15804 tablet.cc:1620] T 4b11ba442cf74fccb4635dd019c0c8d8 P 63331f455be049c1bdd6c23c59ac1bef: MemRowSet was empty: no flush needed.
I20240503 01:59:55.157923 6732 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=72681f43e34d44a484c1b8ddfd8fb6e5] alter complete (version 2)
I20240503 01:59:55.158586 7125 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:55.240866 6882 consensus_queue.cc:237] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:55.242398 7275 raft_consensus.cc:1261] T dcc773c6d5794e26a0003c7d4acc5d21 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:55.242552 9363 raft_consensus.cc:1261] T dcc773c6d5794e26a0003c7d4acc5d21 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:55.242954 14781 consensus_queue.cc:1035] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:55.243752 15668 consensus_queue.cc:1035] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:55.245846 14781 raft_consensus.cc:2932] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:55.246713 7274 raft_consensus.cc:2932] T dcc773c6d5794e26a0003c7d4acc5d21 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:55.247169 9363 raft_consensus.cc:2932] T dcc773c6d5794e26a0003c7d4acc5d21 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:55.251852 6741 catalog_manager.cc:5484] T dcc773c6d5794e26a0003c7d4acc5d21 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:55.255092 7125 tablet_service.cc:1514] Processing DeleteTablet for tablet dcc773c6d5794e26a0003c7d4acc5d21 with delete_type TABLET_DATA_TOMBSTONED (TS c688f3df56ac4b6d9b0c30fa4cc8ca2a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:55.256862 15847 tablet_replica.cc:330] T dcc773c6d5794e26a0003c7d4acc5d21 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:55.257122 15847 raft_consensus.cc:2229] T dcc773c6d5794e26a0003c7d4acc5d21 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:55.257304 15847 raft_consensus.cc:2258] T dcc773c6d5794e26a0003c7d4acc5d21 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:55.257886 15847 ts_tablet_manager.cc:1875] T dcc773c6d5794e26a0003c7d4acc5d21 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:55.261039 15847 ts_tablet_manager.cc:1888] T dcc773c6d5794e26a0003c7d4acc5d21 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 01:59:55.261184 15847 log.cc:1196] T dcc773c6d5794e26a0003c7d4acc5d21 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/dcc773c6d5794e26a0003c7d4acc5d21
I20240503 01:59:55.261394 15731 consensus_queue.cc:1035] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:55.261768 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet dcc773c6d5794e26a0003c7d4acc5d21 (table rebalance_test_table_11 [id=dce2b86d058140cfb0a0d5088d32ec71]) successfully deleted
I20240503 01:59:55.266018 15731 consensus_queue.cc:1035] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:55.270015 6732 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:55.276770 15802 tablet.cc:1722] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:55.277107 15804 tablet.cc:1722] T 4b11ba442cf74fccb4635dd019c0c8d8 P 63331f455be049c1bdd6c23c59ac1bef: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:55.277459 15803 tablet.cc:1722] T 4b11ba442cf74fccb4635dd019c0c8d8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:55.288256 6733 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=72681f43e34d44a484c1b8ddfd8fb6e5] alter complete (version 3)
I20240503 01:59:55.358351 15373 consensus_queue.cc:1035] T 8db6c1313c0042d48416b4f38fe7166f P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:55.358621 15375 consensus_queue.cc:1035] T 8db6c1313c0042d48416b4f38fe7166f P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:55.391947 6737 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:55.392220 6737 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:55.394865 6737 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 4b11ba442cf74fccb4635dd019c0c8d8
I20240503 01:59:55.395524 6993 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b11ba442cf74fccb4635dd019c0c8d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:55 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:55.396128 7125 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b11ba442cf74fccb4635dd019c0c8d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:55 UTC) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:55.396126 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b11ba442cf74fccb4635dd019c0c8d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:55 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:55.396324 15902 tablet_replica.cc:330] T 4b11ba442cf74fccb4635dd019c0c8d8 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:55.396366 15847 tablet_replica.cc:330] T 4b11ba442cf74fccb4635dd019c0c8d8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:55.396591 15847 raft_consensus.cc:2229] T 4b11ba442cf74fccb4635dd019c0c8d8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:55.396809 15847 raft_consensus.cc:2258] T 4b11ba442cf74fccb4635dd019c0c8d8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:55.397115 15903 tablet_replica.cc:330] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:55.398751 15847 ts_tablet_manager.cc:1875] T 4b11ba442cf74fccb4635dd019c0c8d8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:55.399994 15903 raft_consensus.cc:2229] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:55.400965 15903 raft_consensus.cc:2258] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:55.402560 15903 ts_tablet_manager.cc:1875] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:55.404093 15847 ts_tablet_manager.cc:1888] T 4b11ba442cf74fccb4635dd019c0c8d8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:55.404209 15847 log.cc:1196] T 4b11ba442cf74fccb4635dd019c0c8d8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/4b11ba442cf74fccb4635dd019c0c8d8
I20240503 01:59:55.404690 15847 ts_tablet_manager.cc:1909] T 4b11ba442cf74fccb4635dd019c0c8d8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting consensus metadata
I20240503 01:59:55.405207 15902 raft_consensus.cc:2229] T 4b11ba442cf74fccb4635dd019c0c8d8 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:55.405563 15903 ts_tablet_manager.cc:1888] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:55.405653 15903 log.cc:1196] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/4b11ba442cf74fccb4635dd019c0c8d8
I20240503 01:59:55.406128 15903 ts_tablet_manager.cc:1909] T 4b11ba442cf74fccb4635dd019c0c8d8 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:55.406270 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet 4b11ba442cf74fccb4635dd019c0c8d8 (table rebalancer_dynamic_table_new_name [id=72681f43e34d44a484c1b8ddfd8fb6e5]) successfully deleted
I20240503 01:59:55.406735 15902 raft_consensus.cc:2258] T 4b11ba442cf74fccb4635dd019c0c8d8 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:55.407564 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 4b11ba442cf74fccb4635dd019c0c8d8 (table rebalancer_dynamic_table_new_name [id=72681f43e34d44a484c1b8ddfd8fb6e5]) successfully deleted
I20240503 01:59:55.408079 15902 ts_tablet_manager.cc:1875] T 4b11ba442cf74fccb4635dd019c0c8d8 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:55.414381 15902 ts_tablet_manager.cc:1888] T 4b11ba442cf74fccb4635dd019c0c8d8 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:55.414610 15902 log.cc:1196] T 4b11ba442cf74fccb4635dd019c0c8d8 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/4b11ba442cf74fccb4635dd019c0c8d8
I20240503 01:59:55.415165 15902 ts_tablet_manager.cc:1909] T 4b11ba442cf74fccb4635dd019c0c8d8 P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:55.415838 13349 consensus_queue.cc:1035] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:55.416006 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet 4b11ba442cf74fccb4635dd019c0c8d8 (table rebalancer_dynamic_table_new_name [id=72681f43e34d44a484c1b8ddfd8fb6e5]) successfully deleted
I20240503 01:59:55.421618 15247 consensus_queue.cc:1035] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:55.437019 7125 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:55.438549 6993 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:55.447559 6861 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:55.447633 9605 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:55.450455 7256 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:55.466810 9473 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 01:59:55.472667 9735 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:55.474952 9341 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 01:59:55.497977 6737 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:55.505448 9735 tablet_service.cc:1467] Processing CreateTablet for tablet 1e29d7f3486f4f7db480e0cbb279d985 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a3526ff9875441119ac1da4085334035]), partition=
I20240503 01:59:55.505529 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 1e29d7f3486f4f7db480e0cbb279d985 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a3526ff9875441119ac1da4085334035]), partition=
I20240503 01:59:55.505793 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e29d7f3486f4f7db480e0cbb279d985. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:55.505793 9735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e29d7f3486f4f7db480e0cbb279d985. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:55.509296 7125 tablet_service.cc:1467] Processing CreateTablet for tablet 1e29d7f3486f4f7db480e0cbb279d985 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a3526ff9875441119ac1da4085334035]), partition=
I20240503 01:59:55.509570 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e29d7f3486f4f7db480e0cbb279d985. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:55.519444 6737 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:55.525467 15921 tablet_bootstrap.cc:492] T 1e29d7f3486f4f7db480e0cbb279d985 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:55.526187 15922 tablet_bootstrap.cc:492] T 1e29d7f3486f4f7db480e0cbb279d985 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:55.526598 15920 tablet_bootstrap.cc:492] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:55.528966 15921 tablet_bootstrap.cc:654] T 1e29d7f3486f4f7db480e0cbb279d985 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:55.529871 9473 tablet_service.cc:1467] Processing CreateTablet for tablet b69fd5924f9e4d5dae4dac3bf4b9d161 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=bc4eab53df254ef4bc9747cb965654a7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:55.529870 9605 tablet_service.cc:1467] Processing CreateTablet for tablet 6b54d409a37e4a64a00b3c366dc22df7 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=bc4eab53df254ef4bc9747cb965654a7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:55.530194 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b69fd5924f9e4d5dae4dac3bf4b9d161. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:55.530211 9605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b54d409a37e4a64a00b3c366dc22df7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:55.531093 7255 tablet_service.cc:1467] Processing CreateTablet for tablet 8ca4e33409b74f359d3242248652d8bb (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=bc4eab53df254ef4bc9747cb965654a7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:55.531399 7255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ca4e33409b74f359d3242248652d8bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:55.531968 9341 tablet_service.cc:1467] Processing CreateTablet for tablet b69fd5924f9e4d5dae4dac3bf4b9d161 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=bc4eab53df254ef4bc9747cb965654a7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:55.532253 9341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b69fd5924f9e4d5dae4dac3bf4b9d161. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:55.532786 7254 tablet_service.cc:1467] Processing CreateTablet for tablet b69fd5924f9e4d5dae4dac3bf4b9d161 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=bc4eab53df254ef4bc9747cb965654a7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:55.533032 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b69fd5924f9e4d5dae4dac3bf4b9d161. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:55.534686 15925 tablet_bootstrap.cc:492] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:55.535710 15925 tablet_bootstrap.cc:654] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:55.531108 7256 tablet_service.cc:1467] Processing CreateTablet for tablet 6b54d409a37e4a64a00b3c366dc22df7 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=bc4eab53df254ef4bc9747cb965654a7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:55.536334 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b54d409a37e4a64a00b3c366dc22df7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:55.536988 15925 tablet_bootstrap.cc:492] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:55.537077 15925 ts_tablet_manager.cc:1367] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:55.537505 15925 raft_consensus.cc:348] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:55.537638 15925 raft_consensus.cc:374] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:55.537667 15925 raft_consensus.cc:729] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:55.537789 15925 consensus_queue.cc:260] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:55.538062 15925 ts_tablet_manager.cc:1398] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:55.539651 15923 tablet_bootstrap.cc:492] T b69fd5924f9e4d5dae4dac3bf4b9d161 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:55.540412 9606 tablet_service.cc:1467] Processing CreateTablet for tablet 8ca4e33409b74f359d3242248652d8bb (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=bc4eab53df254ef4bc9747cb965654a7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:55.540606 15923 tablet_bootstrap.cc:654] T b69fd5924f9e4d5dae4dac3bf4b9d161 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:55.540693 9606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ca4e33409b74f359d3242248652d8bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:55.542732 15923 tablet_bootstrap.cc:492] T b69fd5924f9e4d5dae4dac3bf4b9d161 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:55.542820 15923 ts_tablet_manager.cc:1367] T b69fd5924f9e4d5dae4dac3bf4b9d161 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:55.543257 15923 raft_consensus.cc:348] T b69fd5924f9e4d5dae4dac3bf4b9d161 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:55.543399 15923 raft_consensus.cc:374] T b69fd5924f9e4d5dae4dac3bf4b9d161 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:55.543442 15923 raft_consensus.cc:729] T b69fd5924f9e4d5dae4dac3bf4b9d161 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:55.543653 15923 consensus_queue.cc:260] T b69fd5924f9e4d5dae4dac3bf4b9d161 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:55.544754 15923 ts_tablet_manager.cc:1398] T b69fd5924f9e4d5dae4dac3bf4b9d161 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:55.542277 7125 tablet_service.cc:1467] Processing CreateTablet for tablet 8ca4e33409b74f359d3242248652d8bb (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=bc4eab53df254ef4bc9747cb965654a7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:55.549198 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ca4e33409b74f359d3242248652d8bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:55.557446 15922 tablet_bootstrap.cc:654] T 1e29d7f3486f4f7db480e0cbb279d985 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:55.560024 15922 tablet_bootstrap.cc:492] T 1e29d7f3486f4f7db480e0cbb279d985 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:55.560711 15922 ts_tablet_manager.cc:1367] T 1e29d7f3486f4f7db480e0cbb279d985 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20240503 01:59:55.530407 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 6b54d409a37e4a64a00b3c366dc22df7 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=bc4eab53df254ef4bc9747cb965654a7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:55.561414 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b54d409a37e4a64a00b3c366dc22df7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:55.562237 15928 tablet_bootstrap.cc:492] T 6b54d409a37e4a64a00b3c366dc22df7 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:55.561614 15922 raft_consensus.cc:348] T 1e29d7f3486f4f7db480e0cbb279d985 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:55.562947 15922 raft_consensus.cc:374] T 1e29d7f3486f4f7db480e0cbb279d985 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:55.562991 15922 raft_consensus.cc:729] T 1e29d7f3486f4f7db480e0cbb279d985 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:55.563213 15922 consensus_queue.cc:260] T 1e29d7f3486f4f7db480e0cbb279d985 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:55.563443 15928 tablet_bootstrap.cc:654] T 6b54d409a37e4a64a00b3c366dc22df7 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:55.563567 15922 ts_tablet_manager.cc:1398] T 1e29d7f3486f4f7db480e0cbb279d985 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:55.563850 15922 tablet_bootstrap.cc:492] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:55.527952 15920 tablet_bootstrap.cc:654] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:55.570510 15922 tablet_bootstrap.cc:654] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:55.571080 15920 tablet_bootstrap.cc:492] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:55.571107 15924 tablet_bootstrap.cc:492] T 8ca4e33409b74f359d3242248652d8bb P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:55.572193 15921 tablet_bootstrap.cc:492] T 1e29d7f3486f4f7db480e0cbb279d985 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:55.572463 15920 ts_tablet_manager.cc:1367] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.000s
I20240503 01:59:55.572489 15922 tablet_bootstrap.cc:492] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:55.572576 15922 ts_tablet_manager.cc:1367] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 01:59:55.573107 15920 raft_consensus.cc:348] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:55.573107 15922 raft_consensus.cc:348] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:55.573343 15922 raft_consensus.cc:374] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:55.573395 15922 raft_consensus.cc:729] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:55.573544 15921 ts_tablet_manager.cc:1367] T 1e29d7f3486f4f7db480e0cbb279d985 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.000s
I20240503 01:59:55.573537 15922 consensus_queue.cc:260] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:55.573904 15922 ts_tablet_manager.cc:1398] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:55.574012 15920 raft_consensus.cc:374] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:55.574108 15920 raft_consensus.cc:729] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:55.574340 15920 consensus_queue.cc:260] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:55.574791 15920 ts_tablet_manager.cc:1398] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:55.577700 15924 tablet_bootstrap.cc:654] T 8ca4e33409b74f359d3242248652d8bb P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:55.579888 15924 tablet_bootstrap.cc:492] T 8ca4e33409b74f359d3242248652d8bb P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:55.580004 15924 ts_tablet_manager.cc:1367] T 8ca4e33409b74f359d3242248652d8bb P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240503 01:59:55.580569 15928 tablet_bootstrap.cc:492] T 6b54d409a37e4a64a00b3c366dc22df7 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:55.580691 15928 ts_tablet_manager.cc:1367] T 6b54d409a37e4a64a00b3c366dc22df7 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 01:59:55.580608 15921 raft_consensus.cc:348] T 1e29d7f3486f4f7db480e0cbb279d985 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:55.580741 15921 raft_consensus.cc:374] T 1e29d7f3486f4f7db480e0cbb279d985 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:55.580811 15921 raft_consensus.cc:729] T 1e29d7f3486f4f7db480e0cbb279d985 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:55.580900 15924 raft_consensus.cc:348] T 8ca4e33409b74f359d3242248652d8bb P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:55.581012 15921 consensus_queue.cc:260] T 1e29d7f3486f4f7db480e0cbb279d985 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:55.581085 15924 raft_consensus.cc:374] T 8ca4e33409b74f359d3242248652d8bb P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:55.581120 15924 raft_consensus.cc:729] T 8ca4e33409b74f359d3242248652d8bb P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:55.581208 15928 raft_consensus.cc:348] T 6b54d409a37e4a64a00b3c366dc22df7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:55.581322 15928 raft_consensus.cc:374] T 6b54d409a37e4a64a00b3c366dc22df7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:55.581364 15928 raft_consensus.cc:729] T 6b54d409a37e4a64a00b3c366dc22df7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:55.581352 15924 consensus_queue.cc:260] T 8ca4e33409b74f359d3242248652d8bb P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:55.581493 15928 consensus_queue.cc:260] T 6b54d409a37e4a64a00b3c366dc22df7 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:55.581542 15921 ts_tablet_manager.cc:1398] T 1e29d7f3486f4f7db480e0cbb279d985 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240503 01:59:55.581728 15921 tablet_bootstrap.cc:492] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:55.581826 15928 ts_tablet_manager.cc:1398] T 6b54d409a37e4a64a00b3c366dc22df7 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:55.582000 15928 tablet_bootstrap.cc:492] T 8ca4e33409b74f359d3242248652d8bb P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:55.582093 15924 ts_tablet_manager.cc:1398] T 8ca4e33409b74f359d3242248652d8bb P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:55.582291 15924 tablet_bootstrap.cc:492] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:55.585434 15924 tablet_bootstrap.cc:654] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:55.586851 15928 tablet_bootstrap.cc:654] T 8ca4e33409b74f359d3242248652d8bb P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:55.587152 15924 tablet_bootstrap.cc:492] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:55.587244 15924 ts_tablet_manager.cc:1367] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:55.587725 15924 raft_consensus.cc:348] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:55.587844 15924 raft_consensus.cc:374] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:55.587877 15924 raft_consensus.cc:729] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:55.587985 15924 consensus_queue.cc:260] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:55.588294 15924 ts_tablet_manager.cc:1398] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:55.588461 15924 tablet_bootstrap.cc:492] T 6b54d409a37e4a64a00b3c366dc22df7 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:55.589206 15924 tablet_bootstrap.cc:654] T 6b54d409a37e4a64a00b3c366dc22df7 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:55.592334 15924 tablet_bootstrap.cc:492] T 6b54d409a37e4a64a00b3c366dc22df7 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:55.592921 15924 ts_tablet_manager.cc:1367] T 6b54d409a37e4a64a00b3c366dc22df7 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:55.593411 15924 raft_consensus.cc:348] T 6b54d409a37e4a64a00b3c366dc22df7 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:55.593536 15924 raft_consensus.cc:374] T 6b54d409a37e4a64a00b3c366dc22df7 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:55.593575 15924 raft_consensus.cc:729] T 6b54d409a37e4a64a00b3c366dc22df7 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:55.593744 15924 consensus_queue.cc:260] T 6b54d409a37e4a64a00b3c366dc22df7 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:55.594287 15924 ts_tablet_manager.cc:1398] T 6b54d409a37e4a64a00b3c366dc22df7 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:55.594337 15921 tablet_bootstrap.cc:654] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:55.597510 15928 tablet_bootstrap.cc:492] T 8ca4e33409b74f359d3242248652d8bb P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:55.597658 15928 ts_tablet_manager.cc:1367] T 8ca4e33409b74f359d3242248652d8bb P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.002s
I20240503 01:59:55.598116 15928 raft_consensus.cc:348] T 8ca4e33409b74f359d3242248652d8bb P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:55.598237 15928 raft_consensus.cc:374] T 8ca4e33409b74f359d3242248652d8bb P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:55.598268 15928 raft_consensus.cc:729] T 8ca4e33409b74f359d3242248652d8bb P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:55.598381 15928 consensus_queue.cc:260] T 8ca4e33409b74f359d3242248652d8bb P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:55.598667 15928 ts_tablet_manager.cc:1398] T 8ca4e33409b74f359d3242248652d8bb P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:55.603121 15921 tablet_bootstrap.cc:492] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:55.603318 15921 ts_tablet_manager.cc:1367] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20240503 01:59:55.603847 15921 raft_consensus.cc:348] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:55.604053 15921 raft_consensus.cc:374] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:55.604135 15921 raft_consensus.cc:729] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:55.604630 15921 consensus_queue.cc:260] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:55.605406 15921 ts_tablet_manager.cc:1398] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:55.618563 7145 consensus_queue.cc:237] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:55.620081 7274 raft_consensus.cc:1261] T 822bafcbc6de406997955ff7a1d9b612 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:55.620214 9494 raft_consensus.cc:1261] T 822bafcbc6de406997955ff7a1d9b612 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:55.620546 15159 consensus_queue.cc:1035] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:55.620779 15163 consensus_queue.cc:1035] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:55.622958 15159 raft_consensus.cc:2932] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:55.623471 9494 raft_consensus.cc:2932] T 822bafcbc6de406997955ff7a1d9b612 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:55.624164 7274 raft_consensus.cc:2932] T 822bafcbc6de406997955ff7a1d9b612 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:55.628150 6737 catalog_manager.cc:5484] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index 4 to 5, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:55.632655 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet 822bafcbc6de406997955ff7a1d9b612 with delete_type TABLET_DATA_TOMBSTONED (TS 5dcc33cc621a4cbbae155f3230fb7c46 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:55.632920 15903 tablet_replica.cc:330] T 822bafcbc6de406997955ff7a1d9b612 P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:55.633071 15903 raft_consensus.cc:2229] T 822bafcbc6de406997955ff7a1d9b612 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:55.633216 15903 raft_consensus.cc:2258] T 822bafcbc6de406997955ff7a1d9b612 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:55.633766 15903 ts_tablet_manager.cc:1875] T 822bafcbc6de406997955ff7a1d9b612 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:55.636194 15903 ts_tablet_manager.cc:1888] T 822bafcbc6de406997955ff7a1d9b612 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 01:59:55.636327 15903 log.cc:1196] T 822bafcbc6de406997955ff7a1d9b612 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/822bafcbc6de406997955ff7a1d9b612
I20240503 01:59:55.637352 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 822bafcbc6de406997955ff7a1d9b612 (table rebalance_test_table_1 [id=38e214942900477190e2c3b67ccc5952]) successfully deleted
I20240503 01:59:55.691828 15668 raft_consensus.cc:482] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:55.692016 15668 raft_consensus.cc:504] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:55.692652 15668 leader_election.cc:288] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:55.693234 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54d409a37e4a64a00b3c366dc22df7" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:55.693418 9625 raft_consensus.cc:2454] T 6b54d409a37e4a64a00b3c366dc22df7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:55.693895 6818 leader_election.cc:302] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:55.693969 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54d409a37e4a64a00b3c366dc22df7" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:55.694151 7274 raft_consensus.cc:2454] T 6b54d409a37e4a64a00b3c366dc22df7 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:55.697607 15668 raft_consensus.cc:2781] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:55.697726 15668 raft_consensus.cc:482] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:55.697768 15668 raft_consensus.cc:3037] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:55.699247 15668 raft_consensus.cc:504] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:55.699707 15668 leader_election.cc:288] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 01:59:55.700201 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54d409a37e4a64a00b3c366dc22df7" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:55.700372 9625 raft_consensus.cc:3037] T 6b54d409a37e4a64a00b3c366dc22df7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:55.700517 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b54d409a37e4a64a00b3c366dc22df7" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:55.700700 7274 raft_consensus.cc:3037] T 6b54d409a37e4a64a00b3c366dc22df7 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:55.701892 7274 raft_consensus.cc:2454] T 6b54d409a37e4a64a00b3c366dc22df7 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:55.702317 9625 raft_consensus.cc:2454] T 6b54d409a37e4a64a00b3c366dc22df7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:55.702456 6817 leader_election.cc:302] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:55.705005 15668 raft_consensus.cc:2781] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:55.705142 15668 raft_consensus.cc:686] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 01:59:55.705322 15668 consensus_queue.cc:237] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:55.708097 6732 catalog_manager.cc:5484] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:55.713485 15383 raft_consensus.cc:482] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:55.713654 15383 raft_consensus.cc:504] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:55.714301 15383 leader_election.cc:288] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:55.715031 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e29d7f3486f4f7db480e0cbb279d985" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:55.715229 6882 raft_consensus.cc:2454] T 1e29d7f3486f4f7db480e0cbb279d985 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 01:59:55.716114 15096 raft_consensus.cc:482] T b69fd5924f9e4d5dae4dac3bf4b9d161 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:55.716238 15096 raft_consensus.cc:504] T b69fd5924f9e4d5dae4dac3bf4b9d161 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:55.717061 15096 leader_election.cc:288] T b69fd5924f9e4d5dae4dac3bf4b9d161 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:55.717957 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b69fd5924f9e4d5dae4dac3bf4b9d161" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:55.718124 7274 raft_consensus.cc:2454] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:55.718683 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b69fd5924f9e4d5dae4dac3bf4b9d161" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:55.718822 9363 raft_consensus.cc:2454] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:55.719121 9428 leader_election.cc:302] T b69fd5924f9e4d5dae4dac3bf4b9d161 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:55.718442 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e29d7f3486f4f7db480e0cbb279d985" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:55.719990 7145 raft_consensus.cc:2454] T 1e29d7f3486f4f7db480e0cbb279d985 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 01:59:55.720661 9692 leader_election.cc:302] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:55.722553 15731 raft_consensus.cc:2781] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:55.722625 15731 raft_consensus.cc:482] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:55.722662 15731 raft_consensus.cc:3037] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:55.723875 15731 raft_consensus.cc:504] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:55.724290 15731 leader_election.cc:288] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Requested vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:55.728360 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e29d7f3486f4f7db480e0cbb279d985" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:55.728534 7145 raft_consensus.cc:3037] T 1e29d7f3486f4f7db480e0cbb279d985 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:55.729166 15330 raft_consensus.cc:2781] T b69fd5924f9e4d5dae4dac3bf4b9d161 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:55.729254 15330 raft_consensus.cc:482] T b69fd5924f9e4d5dae4dac3bf4b9d161 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:55.729297 15330 raft_consensus.cc:3037] T b69fd5924f9e4d5dae4dac3bf4b9d161 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:55.729934 7145 raft_consensus.cc:2454] T 1e29d7f3486f4f7db480e0cbb279d985 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:55.730348 9692 leader_election.cc:302] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:55.730476 15330 raft_consensus.cc:504] T b69fd5924f9e4d5dae4dac3bf4b9d161 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:55.730950 15383 raft_consensus.cc:2781] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:55.730967 15330 leader_election.cc:288] T b69fd5924f9e4d5dae4dac3bf4b9d161 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:55.731030 15383 raft_consensus.cc:686] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Becoming Leader. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Running, Role: LEADER
I20240503 01:59:55.731165 15383 consensus_queue.cc:237] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:55.732486 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b69fd5924f9e4d5dae4dac3bf4b9d161" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:55.732748 9363 raft_consensus.cc:3037] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:55.733800 9363 raft_consensus.cc:2454] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:55.735121 9429 leader_election.cc:302] T b69fd5924f9e4d5dae4dac3bf4b9d161 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:55.736037 15096 raft_consensus.cc:2781] T b69fd5924f9e4d5dae4dac3bf4b9d161 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:55.736068 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b69fd5924f9e4d5dae4dac3bf4b9d161" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:55.736222 7274 raft_consensus.cc:3037] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:55.736127 15096 raft_consensus.cc:686] T b69fd5924f9e4d5dae4dac3bf4b9d161 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Becoming Leader. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Running, Role: LEADER
I20240503 01:59:55.736239 6738 catalog_manager.cc:5484] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: term changed from 0 to 1, leader changed from <none> to db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:55.736405 15096 consensus_queue.cc:237] T b69fd5924f9e4d5dae4dac3bf4b9d161 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:55.737124 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e29d7f3486f4f7db480e0cbb279d985" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:55.737268 6882 raft_consensus.cc:3037] T 1e29d7f3486f4f7db480e0cbb279d985 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:55.737365 7274 raft_consensus.cc:2454] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:55.738355 6882 raft_consensus.cc:2454] T 1e29d7f3486f4f7db480e0cbb279d985 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:55.740289 6740 catalog_manager.cc:5484] T b69fd5924f9e4d5dae4dac3bf4b9d161 P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: term changed from 0 to 1, leader changed from <none> to bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:55.787617 15159 raft_consensus.cc:482] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:55.787801 15159 raft_consensus.cc:504] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:55.788290 15159 leader_election.cc:288] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:55.788924 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca4e33409b74f359d3242248652d8bb" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:55.788864 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca4e33409b74f359d3242248652d8bb" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:55.789116 7274 raft_consensus.cc:2454] T 8ca4e33409b74f359d3242248652d8bb P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:55.789116 9625 raft_consensus.cc:2454] T 8ca4e33409b74f359d3242248652d8bb P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:55.790187 7080 leader_election.cc:302] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:55.791431 15159 raft_consensus.cc:2781] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:55.791561 15159 raft_consensus.cc:482] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:55.791618 15159 raft_consensus.cc:3037] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:55.793327 15159 raft_consensus.cc:504] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:55.793982 15159 leader_election.cc:288] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:55.794942 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca4e33409b74f359d3242248652d8bb" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:55.794958 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca4e33409b74f359d3242248652d8bb" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:55.795105 7274 raft_consensus.cc:3037] T 8ca4e33409b74f359d3242248652d8bb P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:55.795104 9625 raft_consensus.cc:3037] T 8ca4e33409b74f359d3242248652d8bb P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:55.796598 9625 raft_consensus.cc:2454] T 8ca4e33409b74f359d3242248652d8bb P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:55.796912 7274 raft_consensus.cc:2454] T 8ca4e33409b74f359d3242248652d8bb P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:55.797437 7080 leader_election.cc:302] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:55.797703 15159 raft_consensus.cc:2781] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:55.797788 15159 raft_consensus.cc:686] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Becoming Leader. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Running, Role: LEADER
I20240503 01:59:55.797976 15159 consensus_queue.cc:237] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:55.800494 6734 catalog_manager.cc:5484] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:55.825127 7256 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:55.827033 9735 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:55.839161 7125 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 01:59:55.846153 9341 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 01:59:55.847077 6861 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:55.850380 6993 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:55.850705 9473 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:55.882264 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:55.885035 9606 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:55.887239 7145 raft_consensus.cc:1261] T 1e29d7f3486f4f7db480e0cbb279d985 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:55.887704 6882 raft_consensus.cc:1261] T 1e29d7f3486f4f7db480e0cbb279d985 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:55.888217 15383 consensus_queue.cc:1035] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:55.888474 15731 consensus_queue.cc:1035] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:55.891866 6734 catalog_manager.cc:5576] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979) does not have the latest schema for tablet 1e29d7f3486f4f7db480e0cbb279d985 (table rebalancer_dynamic_table [id=a3526ff9875441119ac1da4085334035]). Expected version 1 got 0
I20240503 01:59:55.896189 6734 catalog_manager.cc:5576] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943) does not have the latest schema for tablet 1e29d7f3486f4f7db480e0cbb279d985 (table rebalancer_dynamic_table [id=a3526ff9875441119ac1da4085334035]). Expected version 1 got 0
I20240503 01:59:55.897568 16023 tablet.cc:1722] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:55.898017 16023 tablet.cc:1620] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:55.899744 6740 catalog_manager.cc:5576] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979) does not have the latest schema for tablet 1e29d7f3486f4f7db480e0cbb279d985 (table rebalancer_dynamic_table [id=a3526ff9875441119ac1da4085334035]). Expected version 1 got 0
I20240503 01:59:55.901255 16024 tablet.cc:1722] T 1e29d7f3486f4f7db480e0cbb279d985 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:55.901671 16024 tablet.cc:1620] T 1e29d7f3486f4f7db480e0cbb279d985 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: MemRowSet was empty: no flush needed.
I20240503 01:59:55.902860 6734 catalog_manager.cc:5576] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943) does not have the latest schema for tablet 1e29d7f3486f4f7db480e0cbb279d985 (table rebalancer_dynamic_table [id=a3526ff9875441119ac1da4085334035]). Expected version 1 got 0
I20240503 01:59:55.904942 16025 tablet.cc:1722] T 1e29d7f3486f4f7db480e0cbb279d985 P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:55.905938 16025 tablet.cc:1620] T 1e29d7f3486f4f7db480e0cbb279d985 P 5dcc33cc621a4cbbae155f3230fb7c46: MemRowSet was empty: no flush needed.
I20240503 01:59:55.907909 6734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=a3526ff9875441119ac1da4085334035] alter complete (version 1)
I20240503 01:59:55.974889 9363 consensus_queue.cc:237] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:55.976709 7145 raft_consensus.cc:1261] T 54e3860e5cf54663b4538b9e0f96664e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:55.976909 6882 raft_consensus.cc:1261] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:55.977348 13349 consensus_queue.cc:1035] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:55.977722 15247 consensus_queue.cc:1035] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:55.980126 15917 raft_consensus.cc:2932] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:55.980684 6882 raft_consensus.cc:2932] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:55.982345 7145 raft_consensus.cc:2932] T 54e3860e5cf54663b4538b9e0f96664e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:55.984480 6740 catalog_manager.cc:5484] T 54e3860e5cf54663b4538b9e0f96664e P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:55.989180 9606 tablet_service.cc:1514] Processing DeleteTablet for tablet 54e3860e5cf54663b4538b9e0f96664e with delete_type TABLET_DATA_TOMBSTONED (TS b7062fbbc2b54fff9e99ec4ea5562c76 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:58562
I20240503 01:59:55.990274 16044 tablet_replica.cc:330] T 54e3860e5cf54663b4538b9e0f96664e P b7062fbbc2b54fff9e99ec4ea5562c76: stopping tablet replica
I20240503 01:59:55.990700 16044 raft_consensus.cc:2229] T 54e3860e5cf54663b4538b9e0f96664e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:55.990906 16044 raft_consensus.cc:2258] T 54e3860e5cf54663b4538b9e0f96664e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:55.991701 16044 ts_tablet_manager.cc:1875] T 54e3860e5cf54663b4538b9e0f96664e P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:55.995524 16044 ts_tablet_manager.cc:1888] T 54e3860e5cf54663b4538b9e0f96664e P b7062fbbc2b54fff9e99ec4ea5562c76: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 01:59:55.995841 16044 log.cc:1196] T 54e3860e5cf54663b4538b9e0f96664e P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/54e3860e5cf54663b4538b9e0f96664e
I20240503 01:59:55.996639 6727 catalog_manager.cc:4830] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): tablet 54e3860e5cf54663b4538b9e0f96664e (table rebalancer_extra_table_2 [id=cbc39ac1f52f44debbf6d5a5d0edd3ef]) successfully deleted
I20240503 01:59:56.010934 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:56.018095 16023 tablet.cc:1722] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:56.018493 16023 tablet.cc:1620] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 01:59:56.019852 16024 tablet.cc:1722] T 1e29d7f3486f4f7db480e0cbb279d985 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:56.020181 16024 tablet.cc:1620] T 1e29d7f3486f4f7db480e0cbb279d985 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: MemRowSet was empty: no flush needed.
I20240503 01:59:56.022092 16025 tablet.cc:1722] T 1e29d7f3486f4f7db480e0cbb279d985 P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:56.022425 16025 tablet.cc:1620] T 1e29d7f3486f4f7db480e0cbb279d985 P 5dcc33cc621a4cbbae155f3230fb7c46: MemRowSet was empty: no flush needed.
I20240503 01:59:56.027680 6740 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=a3526ff9875441119ac1da4085334035] alter complete (version 2)
I20240503 01:59:56.062741 9493 consensus_queue.cc:237] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:56.065249 9363 raft_consensus.cc:1261] T 3e824fcc9aac449d9373655b304010e2 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:56.065243 7145 raft_consensus.cc:1261] T 3e824fcc9aac449d9373655b304010e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:56.065788 15330 consensus_queue.cc:1035] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 01:59:56.066082 9430 consensus_peers.cc:485] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a -> Peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): Couldn't send request to peer 5dcc33cc621a4cbbae155f3230fb7c46. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e824fcc9aac449d9373655b304010e2. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:56.066164 15330 consensus_queue.cc:1035] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 01:59:56.069321 15096 raft_consensus.cc:2932] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:56.070094 9363 raft_consensus.cc:2932] T 3e824fcc9aac449d9373655b304010e2 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:56.075227 6734 catalog_manager.cc:5484] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:56.079191 7145 raft_consensus.cc:2932] T 3e824fcc9aac449d9373655b304010e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:56.131844 6736 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:56.137995 16023 tablet.cc:1722] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:56.139385 16024 tablet.cc:1722] T 1e29d7f3486f4f7db480e0cbb279d985 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:56.140841 16025 tablet.cc:1722] T 1e29d7f3486f4f7db480e0cbb279d985 P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:56.145836 15330 consensus_queue.cc:1035] T b69fd5924f9e4d5dae4dac3bf4b9d161 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:56.149644 6734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=a3526ff9875441119ac1da4085334035] alter complete (version 3)
I20240503 01:59:56.151482 14781 consensus_queue.cc:1035] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:56.152297 15096 consensus_queue.cc:1035] T b69fd5924f9e4d5dae4dac3bf4b9d161 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:56.156495 15668 consensus_queue.cc:1035] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:56.246403 15159 consensus_queue.cc:1035] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:56.258926 15957 consensus_queue.cc:1035] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:56.263500 6733 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:56.263689 6733 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:56.266153 6733 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 1e29d7f3486f4f7db480e0cbb279d985
I20240503 01:59:56.267024 9735 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e29d7f3486f4f7db480e0cbb279d985 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:56 UTC) from {username='slave'} at 127.0.0.1:48152
I20240503 01:59:56.267427 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e29d7f3486f4f7db480e0cbb279d985 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:56 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:56.268436 7125 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e29d7f3486f4f7db480e0cbb279d985 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:56 UTC) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:56.268812 16137 tablet_replica.cc:330] T 1e29d7f3486f4f7db480e0cbb279d985 P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:56.268805 16147 tablet_replica.cc:330] T 1e29d7f3486f4f7db480e0cbb279d985 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:56.268963 16147 raft_consensus.cc:2229] T 1e29d7f3486f4f7db480e0cbb279d985 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:56.268963 16137 raft_consensus.cc:2229] T 1e29d7f3486f4f7db480e0cbb279d985 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:56.269090 16147 raft_consensus.cc:2258] T 1e29d7f3486f4f7db480e0cbb279d985 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:56.269091 16137 raft_consensus.cc:2258] T 1e29d7f3486f4f7db480e0cbb279d985 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:56.269886 16137 ts_tablet_manager.cc:1875] T 1e29d7f3486f4f7db480e0cbb279d985 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:56.270264 16147 ts_tablet_manager.cc:1875] T 1e29d7f3486f4f7db480e0cbb279d985 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:56.273309 16137 ts_tablet_manager.cc:1888] T 1e29d7f3486f4f7db480e0cbb279d985 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:56.273506 16137 log.cc:1196] T 1e29d7f3486f4f7db480e0cbb279d985 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/1e29d7f3486f4f7db480e0cbb279d985
I20240503 01:59:56.273929 16137 ts_tablet_manager.cc:1909] T 1e29d7f3486f4f7db480e0cbb279d985 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:56.275146 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 1e29d7f3486f4f7db480e0cbb279d985 (table rebalancer_dynamic_table_new_name [id=a3526ff9875441119ac1da4085334035]) successfully deleted
I20240503 01:59:56.275823 16147 ts_tablet_manager.cc:1888] T 1e29d7f3486f4f7db480e0cbb279d985 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:56.275964 16147 log.cc:1196] T 1e29d7f3486f4f7db480e0cbb279d985 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/1e29d7f3486f4f7db480e0cbb279d985
I20240503 01:59:56.276481 16147 ts_tablet_manager.cc:1909] T 1e29d7f3486f4f7db480e0cbb279d985 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting consensus metadata
I20240503 01:59:56.280822 16136 tablet_replica.cc:330] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191: stopping tablet replica
I20240503 01:59:56.281972 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet 1e29d7f3486f4f7db480e0cbb279d985 (table rebalancer_dynamic_table_new_name [id=a3526ff9875441119ac1da4085334035]) successfully deleted
I20240503 01:59:56.283918 9341 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 01:59:56.284242 7256 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:56.304394 6861 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:56.307845 6993 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:56.325320 16136 raft_consensus.cc:2229] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:56.325632 16136 raft_consensus.cc:2258] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:56.326088 9735 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:56.329500 16136 ts_tablet_manager.cc:1875] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:56.332248 9606 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:56.342803 9473 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:56.349802 16136 ts_tablet_manager.cc:1888] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:56.349915 16136 log.cc:1196] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/1e29d7f3486f4f7db480e0cbb279d985
I20240503 01:59:56.350224 16136 ts_tablet_manager.cc:1909] T 1e29d7f3486f4f7db480e0cbb279d985 P db1a0000aaa346f79c0aa4b6c5105191: Deleting consensus metadata
I20240503 01:59:56.351266 6733 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:56.353844 7125 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 01:59:56.363965 9735 tablet_service.cc:1467] Processing CreateTablet for tablet e8cb37039be049cf9c4e53bbcfbd80d9 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=b11be10400574bd0a4c4768ad34c1fc6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:56.364295 9735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8cb37039be049cf9c4e53bbcfbd80d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.364486 9606 tablet_service.cc:1467] Processing CreateTablet for tablet 4a6a5ff679364e2f808bfd5f19b25fda (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=b11be10400574bd0a4c4768ad34c1fc6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:56.364925 9606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a6a5ff679364e2f808bfd5f19b25fda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.366634 9341 tablet_service.cc:1467] Processing CreateTablet for tablet 4a6a5ff679364e2f808bfd5f19b25fda (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=b11be10400574bd0a4c4768ad34c1fc6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:56.366950 9341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a6a5ff679364e2f808bfd5f19b25fda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.367081 9604 tablet_service.cc:1467] Processing CreateTablet for tablet 1b7d65d849e242b4b6c0bc06422001db (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=b11be10400574bd0a4c4768ad34c1fc6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:56.367326 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b7d65d849e242b4b6c0bc06422001db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.368237 16151 tablet_bootstrap.cc:492] T 4a6a5ff679364e2f808bfd5f19b25fda P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:56.368217 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 1b7d65d849e242b4b6c0bc06422001db (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=b11be10400574bd0a4c4768ad34c1fc6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:56.368497 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b7d65d849e242b4b6c0bc06422001db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.370483 6992 tablet_service.cc:1467] Processing CreateTablet for tablet e8cb37039be049cf9c4e53bbcfbd80d9 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=b11be10400574bd0a4c4768ad34c1fc6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:56.370952 6992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8cb37039be049cf9c4e53bbcfbd80d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.371107 9605 tablet_service.cc:1467] Processing CreateTablet for tablet e8cb37039be049cf9c4e53bbcfbd80d9 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=b11be10400574bd0a4c4768ad34c1fc6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:56.371485 9605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8cb37039be049cf9c4e53bbcfbd80d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.371336 6733 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:56.373679 6993 tablet_service.cc:1467] Processing CreateTablet for tablet 4a6a5ff679364e2f808bfd5f19b25fda (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=b11be10400574bd0a4c4768ad34c1fc6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:56.373950 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a6a5ff679364e2f808bfd5f19b25fda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.374806 6994 tablet_service.cc:1467] Processing CreateTablet for tablet 1b7d65d849e242b4b6c0bc06422001db (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=b11be10400574bd0a4c4768ad34c1fc6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:56.375034 6994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b7d65d849e242b4b6c0bc06422001db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.388855 9473 tablet_service.cc:1467] Processing CreateTablet for tablet 1443f81724f74fc99cf10616032f24ac (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c0055f5131754db8beafabfa1f12d875]), partition=
I20240503 01:59:56.389183 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1443f81724f74fc99cf10616032f24ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.391489 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 1443f81724f74fc99cf10616032f24ac (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c0055f5131754db8beafabfa1f12d875]), partition=
I20240503 01:59:56.392097 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1443f81724f74fc99cf10616032f24ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.396754 7125 tablet_service.cc:1467] Processing CreateTablet for tablet 1443f81724f74fc99cf10616032f24ac (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c0055f5131754db8beafabfa1f12d875]), partition=
I20240503 01:59:56.397145 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1443f81724f74fc99cf10616032f24ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.399137 16151 tablet_bootstrap.cc:654] T 4a6a5ff679364e2f808bfd5f19b25fda P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:56.403290 16151 tablet_bootstrap.cc:492] T 4a6a5ff679364e2f808bfd5f19b25fda P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:56.404783 16154 tablet_bootstrap.cc:492] T 1b7d65d849e242b4b6c0bc06422001db P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:56.405285 16151 ts_tablet_manager.cc:1367] T 4a6a5ff679364e2f808bfd5f19b25fda P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.037s user 0.000s sys 0.003s
I20240503 01:59:56.406291 16154 tablet_bootstrap.cc:654] T 1b7d65d849e242b4b6c0bc06422001db P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:56.409101 16151 raft_consensus.cc:348] T 4a6a5ff679364e2f808bfd5f19b25fda P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:56.409540 16151 raft_consensus.cc:374] T 4a6a5ff679364e2f808bfd5f19b25fda P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:56.409667 16151 raft_consensus.cc:729] T 4a6a5ff679364e2f808bfd5f19b25fda P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:56.409901 16151 consensus_queue.cc:260] T 4a6a5ff679364e2f808bfd5f19b25fda P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:56.410650 16151 ts_tablet_manager.cc:1398] T 4a6a5ff679364e2f808bfd5f19b25fda P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240503 01:59:56.410912 16151 tablet_bootstrap.cc:492] T 1b7d65d849e242b4b6c0bc06422001db P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:56.412401 16151 tablet_bootstrap.cc:654] T 1b7d65d849e242b4b6c0bc06422001db P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:56.413839 16154 tablet_bootstrap.cc:492] T 1b7d65d849e242b4b6c0bc06422001db P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:56.413838 16151 tablet_bootstrap.cc:492] T 1b7d65d849e242b4b6c0bc06422001db P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:56.413954 16154 ts_tablet_manager.cc:1367] T 1b7d65d849e242b4b6c0bc06422001db P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240503 01:59:56.413976 16151 ts_tablet_manager.cc:1367] T 1b7d65d849e242b4b6c0bc06422001db P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:56.414810 16151 raft_consensus.cc:348] T 1b7d65d849e242b4b6c0bc06422001db P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:56.415174 16151 raft_consensus.cc:374] T 1b7d65d849e242b4b6c0bc06422001db P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:56.415115 16154 raft_consensus.cc:348] T 1b7d65d849e242b4b6c0bc06422001db P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:56.415217 16151 raft_consensus.cc:729] T 1b7d65d849e242b4b6c0bc06422001db P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:56.415234 16154 raft_consensus.cc:374] T 1b7d65d849e242b4b6c0bc06422001db P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:56.415269 16154 raft_consensus.cc:729] T 1b7d65d849e242b4b6c0bc06422001db P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:56.415374 16151 consensus_queue.cc:260] T 1b7d65d849e242b4b6c0bc06422001db P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:56.415401 16154 consensus_queue.cc:260] T 1b7d65d849e242b4b6c0bc06422001db P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:56.415992 16151 ts_tablet_manager.cc:1398] T 1b7d65d849e242b4b6c0bc06422001db P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:56.415992 16154 ts_tablet_manager.cc:1398] T 1b7d65d849e242b4b6c0bc06422001db P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:56.416625 16154 tablet_bootstrap.cc:492] T 1443f81724f74fc99cf10616032f24ac P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:56.418193 16154 tablet_bootstrap.cc:654] T 1443f81724f74fc99cf10616032f24ac P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:56.418648 16151 tablet_bootstrap.cc:492] T e8cb37039be049cf9c4e53bbcfbd80d9 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:56.420431 16151 tablet_bootstrap.cc:654] T e8cb37039be049cf9c4e53bbcfbd80d9 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:56.421330 16154 tablet_bootstrap.cc:492] T 1443f81724f74fc99cf10616032f24ac P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:56.421669 16154 ts_tablet_manager.cc:1367] T 1443f81724f74fc99cf10616032f24ac P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240503 01:59:56.422461 16154 raft_consensus.cc:348] T 1443f81724f74fc99cf10616032f24ac P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:56.422711 16154 raft_consensus.cc:374] T 1443f81724f74fc99cf10616032f24ac P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:56.422778 16154 raft_consensus.cc:729] T 1443f81724f74fc99cf10616032f24ac P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:56.422847 6726 catalog_manager.cc:4830] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): tablet 1e29d7f3486f4f7db480e0cbb279d985 (table rebalancer_dynamic_table_new_name [id=a3526ff9875441119ac1da4085334035]) successfully deleted
I20240503 01:59:56.422937 16154 consensus_queue.cc:260] T 1443f81724f74fc99cf10616032f24ac P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:56.423425 16154 ts_tablet_manager.cc:1398] T 1443f81724f74fc99cf10616032f24ac P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:56.431147 16150 tablet_bootstrap.cc:492] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:56.432468 16150 tablet_bootstrap.cc:654] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:56.433599 16157 tablet_bootstrap.cc:492] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:56.433871 16152 tablet_bootstrap.cc:492] T 4a6a5ff679364e2f808bfd5f19b25fda P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:56.434121 16155 tablet_bootstrap.cc:492] T 1443f81724f74fc99cf10616032f24ac P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:56.434319 16153 tablet_bootstrap.cc:492] T e8cb37039be049cf9c4e53bbcfbd80d9 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:56.434872 16150 tablet_bootstrap.cc:492] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:56.434983 16150 ts_tablet_manager.cc:1367] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 01:59:56.435478 16150 raft_consensus.cc:348] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:56.435649 16150 raft_consensus.cc:374] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:56.435699 16150 raft_consensus.cc:729] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:56.435828 16150 consensus_queue.cc:260] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:56.436139 16150 ts_tablet_manager.cc:1398] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:56.437319 16157 tablet_bootstrap.cc:654] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:56.440007 16153 tablet_bootstrap.cc:654] T e8cb37039be049cf9c4e53bbcfbd80d9 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:56.440409 16155 tablet_bootstrap.cc:654] T 1443f81724f74fc99cf10616032f24ac P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:56.442963 16152 tablet_bootstrap.cc:654] T 4a6a5ff679364e2f808bfd5f19b25fda P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:56.444222 16152 tablet_bootstrap.cc:492] T 4a6a5ff679364e2f808bfd5f19b25fda P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:56.444389 16152 ts_tablet_manager.cc:1367] T 4a6a5ff679364e2f808bfd5f19b25fda P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240503 01:59:56.444972 16152 raft_consensus.cc:348] T 4a6a5ff679364e2f808bfd5f19b25fda P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:56.445181 16152 raft_consensus.cc:374] T 4a6a5ff679364e2f808bfd5f19b25fda P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:56.445271 16152 raft_consensus.cc:729] T 4a6a5ff679364e2f808bfd5f19b25fda P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:56.446018 16152 consensus_queue.cc:260] T 4a6a5ff679364e2f808bfd5f19b25fda P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:56.447791 16151 tablet_bootstrap.cc:492] T e8cb37039be049cf9c4e53bbcfbd80d9 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:56.447921 16151 ts_tablet_manager.cc:1367] T e8cb37039be049cf9c4e53bbcfbd80d9 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20240503 01:59:56.448437 16151 raft_consensus.cc:348] T e8cb37039be049cf9c4e53bbcfbd80d9 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:56.448575 16151 raft_consensus.cc:374] T e8cb37039be049cf9c4e53bbcfbd80d9 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:56.448604 16151 raft_consensus.cc:729] T e8cb37039be049cf9c4e53bbcfbd80d9 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:56.448724 16151 consensus_queue.cc:260] T e8cb37039be049cf9c4e53bbcfbd80d9 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:56.448994 16151 ts_tablet_manager.cc:1398] T e8cb37039be049cf9c4e53bbcfbd80d9 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:56.450881 16152 ts_tablet_manager.cc:1398] T 4a6a5ff679364e2f808bfd5f19b25fda P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 01:59:56.452596 16155 tablet_bootstrap.cc:492] T 1443f81724f74fc99cf10616032f24ac P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:56.452719 16155 ts_tablet_manager.cc:1367] T 1443f81724f74fc99cf10616032f24ac P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240503 01:59:56.453202 16155 raft_consensus.cc:348] T 1443f81724f74fc99cf10616032f24ac P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:56.453331 16155 raft_consensus.cc:374] T 1443f81724f74fc99cf10616032f24ac P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:56.453363 16155 raft_consensus.cc:729] T 1443f81724f74fc99cf10616032f24ac P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:56.453584 16155 consensus_queue.cc:260] T 1443f81724f74fc99cf10616032f24ac P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:56.455287 16157 tablet_bootstrap.cc:492] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:56.455412 16157 ts_tablet_manager.cc:1367] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240503 01:59:56.454689 16153 tablet_bootstrap.cc:492] T e8cb37039be049cf9c4e53bbcfbd80d9 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:56.455958 16153 ts_tablet_manager.cc:1367] T e8cb37039be049cf9c4e53bbcfbd80d9 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20240503 01:59:56.456477 16153 raft_consensus.cc:348] T e8cb37039be049cf9c4e53bbcfbd80d9 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:56.456597 16153 raft_consensus.cc:374] T e8cb37039be049cf9c4e53bbcfbd80d9 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:56.456629 16153 raft_consensus.cc:729] T e8cb37039be049cf9c4e53bbcfbd80d9 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:56.456779 16153 consensus_queue.cc:260] T e8cb37039be049cf9c4e53bbcfbd80d9 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:56.457082 16153 ts_tablet_manager.cc:1398] T e8cb37039be049cf9c4e53bbcfbd80d9 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:56.457267 16157 raft_consensus.cc:348] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:56.457376 16157 raft_consensus.cc:374] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:56.457415 16157 raft_consensus.cc:729] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:56.457583 16157 consensus_queue.cc:260] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:56.457882 16157 ts_tablet_manager.cc:1398] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:56.458129 16155 ts_tablet_manager.cc:1398] T 1443f81724f74fc99cf10616032f24ac P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:56.458642 16153 tablet_bootstrap.cc:492] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:56.461886 16153 tablet_bootstrap.cc:654] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:56.463936 16153 tablet_bootstrap.cc:492] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:56.464080 16153 ts_tablet_manager.cc:1367] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240503 01:59:56.464836 16153 raft_consensus.cc:348] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:56.465013 16153 raft_consensus.cc:374] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:56.465058 16153 raft_consensus.cc:729] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:56.465216 16153 consensus_queue.cc:260] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:56.465775 16153 ts_tablet_manager.cc:1398] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:56.466007 16153 tablet_bootstrap.cc:492] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:56.467614 16153 tablet_bootstrap.cc:654] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:56.469389 16153 tablet_bootstrap.cc:492] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:56.469508 16153 ts_tablet_manager.cc:1367] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 01:59:56.470257 16153 raft_consensus.cc:348] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:56.470413 16153 raft_consensus.cc:374] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:56.470453 16153 raft_consensus.cc:729] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:56.470598 16153 consensus_queue.cc:260] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:56.470916 16153 ts_tablet_manager.cc:1398] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:56.504704 7012 consensus_queue.cc:237] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:56.506507 9755 raft_consensus.cc:1261] T fb287629baab4cada193ae57695ef1b5 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:56.506636 9493 raft_consensus.cc:1261] T fb287629baab4cada193ae57695ef1b5 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:56.507014 14833 consensus_queue.cc:1035] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:56.507375 14833 consensus_queue.cc:1035] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:56.509291 16187 raft_consensus.cc:2932] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:56.510048 9493 raft_consensus.cc:2932] T fb287629baab4cada193ae57695ef1b5 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:56.512934 9755 raft_consensus.cc:2932] T fb287629baab4cada193ae57695ef1b5 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:56.513640 6741 catalog_manager.cc:5484] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:56.522550 9605 tablet_service.cc:1514] Processing DeleteTablet for tablet fb287629baab4cada193ae57695ef1b5 with delete_type TABLET_DATA_TOMBSTONED (TS b7062fbbc2b54fff9e99ec4ea5562c76 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:58562
I20240503 01:59:56.524243 16190 tablet_replica.cc:330] T fb287629baab4cada193ae57695ef1b5 P b7062fbbc2b54fff9e99ec4ea5562c76: stopping tablet replica
I20240503 01:59:56.524935 15957 raft_consensus.cc:482] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:56.525056 15957 raft_consensus.cc:504] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:56.525285 16190 raft_consensus.cc:2229] T fb287629baab4cada193ae57695ef1b5 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:56.525455 16190 raft_consensus.cc:2258] T fb287629baab4cada193ae57695ef1b5 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:56.525633 15957 leader_election.cc:288] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:56.526008 16190 ts_tablet_manager.cc:1875] T fb287629baab4cada193ae57695ef1b5 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:56.526052 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1443f81724f74fc99cf10616032f24ac" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:56.526188 6881 raft_consensus.cc:2454] T 1443f81724f74fc99cf10616032f24ac P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:56.526290 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1443f81724f74fc99cf10616032f24ac" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:56.526435 9493 raft_consensus.cc:2454] T 1443f81724f74fc99cf10616032f24ac P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:56.526546 7081 leader_election.cc:302] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:56.527017 15957 raft_consensus.cc:2781] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:56.527097 15957 raft_consensus.cc:482] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:56.527138 15957 raft_consensus.cc:3037] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:56.528244 15957 raft_consensus.cc:504] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:56.528656 15957 leader_election.cc:288] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Requested vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:56.529089 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1443f81724f74fc99cf10616032f24ac" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:56.529096 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1443f81724f74fc99cf10616032f24ac" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:56.529217 6881 raft_consensus.cc:3037] T 1443f81724f74fc99cf10616032f24ac P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:56.529227 9493 raft_consensus.cc:3037] T 1443f81724f74fc99cf10616032f24ac P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:56.530206 16190 ts_tablet_manager.cc:1888] T fb287629baab4cada193ae57695ef1b5 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 01:59:56.530261 6881 raft_consensus.cc:2454] T 1443f81724f74fc99cf10616032f24ac P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:56.530287 16190 log.cc:1196] T fb287629baab4cada193ae57695ef1b5 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/fb287629baab4cada193ae57695ef1b5
I20240503 01:59:56.530275 9493 raft_consensus.cc:2454] T 1443f81724f74fc99cf10616032f24ac P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:56.530926 6727 catalog_manager.cc:4830] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): tablet fb287629baab4cada193ae57695ef1b5 (table rebalancer_extra_table_2 [id=cbc39ac1f52f44debbf6d5a5d0edd3ef]) successfully deleted
I20240503 01:59:56.531199 7078 leader_election.cc:302] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:56.531383 15957 raft_consensus.cc:2781] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:56.531472 15957 raft_consensus.cc:686] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Becoming Leader. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Running, Role: LEADER
I20240503 01:59:56.531621 15957 consensus_queue.cc:237] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:56.535109 6735 catalog_manager.cc:5484] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:56.613270 16187 raft_consensus.cc:482] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:56.613430 16187 raft_consensus.cc:504] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:56.613884 16187 leader_election.cc:288] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:56.614758 9624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7d65d849e242b4b6c0bc06422001db" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:56.614913 9624 raft_consensus.cc:2454] T 1b7d65d849e242b4b6c0bc06422001db P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:56.615152 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7d65d849e242b4b6c0bc06422001db" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:56.615283 6881 raft_consensus.cc:2454] T 1b7d65d849e242b4b6c0bc06422001db P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:56.615857 6950 leader_election.cc:302] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:56.616451 16187 raft_consensus.cc:2781] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:56.616514 16187 raft_consensus.cc:482] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:56.616557 16187 raft_consensus.cc:3037] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:56.618245 16187 raft_consensus.cc:504] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:56.618739 16187 leader_election.cc:288] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:56.619542 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7d65d849e242b4b6c0bc06422001db" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:56.619684 6881 raft_consensus.cc:3037] T 1b7d65d849e242b4b6c0bc06422001db P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:56.621050 6881 raft_consensus.cc:2454] T 1b7d65d849e242b4b6c0bc06422001db P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:56.621428 6950 leader_election.cc:302] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:56.623010 9624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7d65d849e242b4b6c0bc06422001db" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:56.623256 9624 raft_consensus.cc:3037] T 1b7d65d849e242b4b6c0bc06422001db P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:56.624616 9624 raft_consensus.cc:2454] T 1b7d65d849e242b4b6c0bc06422001db P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:56.624974 15227 raft_consensus.cc:2781] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:56.625085 15227 raft_consensus.cc:686] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:56.625232 15227 consensus_queue.cc:237] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:56.635422 6738 catalog_manager.cc:5484] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:56.642482 16231 raft_consensus.cc:482] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:56.642685 16231 raft_consensus.cc:504] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:56.643239 16231 leader_election.cc:288] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:56.643561 9624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8cb37039be049cf9c4e53bbcfbd80d9" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:56.643711 9624 raft_consensus.cc:2454] T e8cb37039be049cf9c4e53bbcfbd80d9 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 01:59:56.644019 9692 leader_election.cc:302] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:56.644233 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8cb37039be049cf9c4e53bbcfbd80d9" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:56.644371 7012 raft_consensus.cc:2454] T e8cb37039be049cf9c4e53bbcfbd80d9 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 01:59:56.645387 16231 raft_consensus.cc:2781] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:56.645452 16231 raft_consensus.cc:482] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:56.645490 16231 raft_consensus.cc:3037] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:56.647418 6737 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:56.648871 16231 raft_consensus.cc:504] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:56.649782 9624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8cb37039be049cf9c4e53bbcfbd80d9" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:56.649914 9624 raft_consensus.cc:3037] T e8cb37039be049cf9c4e53bbcfbd80d9 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:56.650931 9624 raft_consensus.cc:2454] T e8cb37039be049cf9c4e53bbcfbd80d9 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:56.651587 9692 leader_election.cc:302] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:56.651826 15731 raft_consensus.cc:2781] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:56.651908 15731 raft_consensus.cc:686] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Becoming Leader. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Running, Role: LEADER
I20240503 01:59:56.652035 15731 consensus_queue.cc:237] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:56.652981 9493 raft_consensus.cc:1261] T 1443f81724f74fc99cf10616032f24ac P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:56.653321 16231 leader_election.cc:288] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:56.653403 15159 consensus_queue.cc:1035] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:56.654336 6881 raft_consensus.cc:1261] T 1443f81724f74fc99cf10616032f24ac P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:56.657187 6738 catalog_manager.cc:5576] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471) does not have the latest schema for tablet 1443f81724f74fc99cf10616032f24ac (table rebalancer_dynamic_table [id=c0055f5131754db8beafabfa1f12d875]). Expected version 1 got 0
I20240503 01:59:56.659550 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8cb37039be049cf9c4e53bbcfbd80d9" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:56.659754 7012 raft_consensus.cc:3037] T e8cb37039be049cf9c4e53bbcfbd80d9 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:56.659765 15957 consensus_queue.cc:1035] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:56.660586 16236 tablet.cc:1722] T 1443f81724f74fc99cf10616032f24ac P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:56.660957 7012 raft_consensus.cc:2454] T e8cb37039be049cf9c4e53bbcfbd80d9 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 01:59:56.661059 16236 tablet.cc:1620] T 1443f81724f74fc99cf10616032f24ac P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:56.663183 16237 mvcc.cc:205] Tried to move back new op lower bound from 7023417739886555136 to 7023417739394871296. Current Snapshot: MvccSnapshot[applied={T|T < 7023417739886555136}]
I20240503 01:59:56.664090 16239 tablet.cc:1722] T 1443f81724f74fc99cf10616032f24ac P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:56.664410 16239 tablet.cc:1620] T 1443f81724f74fc99cf10616032f24ac P 5dcc33cc621a4cbbae155f3230fb7c46: MemRowSet was empty: no flush needed.
I20240503 01:59:56.665438 6733 catalog_manager.cc:5576] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943) does not have the latest schema for tablet 1443f81724f74fc99cf10616032f24ac (table rebalancer_dynamic_table [id=c0055f5131754db8beafabfa1f12d875]). Expected version 1 got 0
I20240503 01:59:56.668473 6734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=c0055f5131754db8beafabfa1f12d875] alter complete (version 1)
I20240503 01:59:56.669356 6738 catalog_manager.cc:5484] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: term changed from 0 to 1, leader changed from <none> to db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:56.673846 16235 tablet.cc:1722] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:56.674213 16235 tablet.cc:1620] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a: MemRowSet was empty: no flush needed.
I20240503 01:59:56.692365 16240 ts_tablet_manager.cc:902] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46: Initiating tablet copy from peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:56.693140 16240 tablet_copy_client.cc:308] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:43471
I20240503 01:59:56.693863 9514 tablet_copy_service.cc:140] P bb5364d730cb4489a0bfc6c270b7450a: Received BeginTabletCopySession request for tablet 3e824fcc9aac449d9373655b304010e2 from peer 5dcc33cc621a4cbbae155f3230fb7c46 ({username='slave'} at 127.0.99.129:38243)
I20240503 01:59:56.694007 9514 tablet_copy_service.cc:161] P bb5364d730cb4489a0bfc6c270b7450a: Beginning new tablet copy session on tablet 3e824fcc9aac449d9373655b304010e2 from peer 5dcc33cc621a4cbbae155f3230fb7c46 at {username='slave'} at 127.0.99.129:38243: session id = 5dcc33cc621a4cbbae155f3230fb7c46-3e824fcc9aac449d9373655b304010e2
I20240503 01:59:56.695082 9514 tablet_copy_source_session.cc:205] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:56.696480 16240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e824fcc9aac449d9373655b304010e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:56.698962 16240 tablet_copy_client.cc:783] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:56.699178 16240 tablet_copy_client.cc:647] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:56.700569 16240 tablet_copy_client.cc:523] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:56.702096 16240 tablet_bootstrap.cc:492] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:56.705422 16240 tablet_bootstrap.cc:492] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:56.705670 16240 tablet_bootstrap.cc:492] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap complete.
I20240503 01:59:56.705852 16240 ts_tablet_manager.cc:1367] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20240503 01:59:56.706272 16240 raft_consensus.cc:348] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:56.706384 16240 raft_consensus.cc:729] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:56.706563 16240 consensus_queue.cc:260] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:56.706840 16240 ts_tablet_manager.cc:1398] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:56.707454 9514 tablet_copy_service.cc:342] P bb5364d730cb4489a0bfc6c270b7450a: Request end of tablet copy session 5dcc33cc621a4cbbae155f3230fb7c46-3e824fcc9aac449d9373655b304010e2 received from {username='slave'} at 127.0.99.129:38243
I20240503 01:59:56.707556 9514 tablet_copy_service.cc:433] P bb5364d730cb4489a0bfc6c270b7450a: ending tablet copy session 5dcc33cc621a4cbbae155f3230fb7c46-3e824fcc9aac449d9373655b304010e2 on tablet 3e824fcc9aac449d9373655b304010e2 with peer 5dcc33cc621a4cbbae155f3230fb7c46
I20240503 01:59:56.772243 6739 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:56.786345 16239 tablet.cc:1722] T 1443f81724f74fc99cf10616032f24ac P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:56.786994 16235 tablet.cc:1722] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:56.787097 16239 tablet.cc:1620] T 1443f81724f74fc99cf10616032f24ac P 5dcc33cc621a4cbbae155f3230fb7c46: MemRowSet was empty: no flush needed.
I20240503 01:59:56.787329 16235 tablet.cc:1620] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a: MemRowSet was empty: no flush needed.
I20240503 01:59:56.793525 6734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=c0055f5131754db8beafabfa1f12d875] alter complete (version 2)
I20240503 01:59:56.794253 16236 tablet.cc:1722] T 1443f81724f74fc99cf10616032f24ac P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:56.794616 16236 tablet.cc:1620] T 1443f81724f74fc99cf10616032f24ac P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:56.842025 16102 raft_consensus.cc:482] T 4a6a5ff679364e2f808bfd5f19b25fda P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:56.842164 16102 raft_consensus.cc:504] T 4a6a5ff679364e2f808bfd5f19b25fda P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:56.842746 16102 leader_election.cc:288] T 4a6a5ff679364e2f808bfd5f19b25fda P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:56.843158 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6a5ff679364e2f808bfd5f19b25fda" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:56.843327 7012 raft_consensus.cc:2454] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 01:59:56.843300 9361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6a5ff679364e2f808bfd5f19b25fda" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:56.843441 9361 raft_consensus.cc:2454] T 4a6a5ff679364e2f808bfd5f19b25fda P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 01:59:56.843730 9559 leader_election.cc:302] T 4a6a5ff679364e2f808bfd5f19b25fda P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:56.843979 16102 raft_consensus.cc:2781] T 4a6a5ff679364e2f808bfd5f19b25fda P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:56.844028 16102 raft_consensus.cc:482] T 4a6a5ff679364e2f808bfd5f19b25fda P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:56.844064 16102 raft_consensus.cc:3037] T 4a6a5ff679364e2f808bfd5f19b25fda P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:56.845429 16102 raft_consensus.cc:504] T 4a6a5ff679364e2f808bfd5f19b25fda P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:56.846011 16102 leader_election.cc:288] T 4a6a5ff679364e2f808bfd5f19b25fda P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Requested vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:56.847270 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6a5ff679364e2f808bfd5f19b25fda" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:56.847436 7012 raft_consensus.cc:3037] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:56.847621 9361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6a5ff679364e2f808bfd5f19b25fda" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:56.847738 9361 raft_consensus.cc:3037] T 4a6a5ff679364e2f808bfd5f19b25fda P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:56.848558 7012 raft_consensus.cc:2454] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 01:59:56.848942 9361 raft_consensus.cc:2454] T 4a6a5ff679364e2f808bfd5f19b25fda P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 01:59:56.849014 9559 leader_election.cc:302] T 4a6a5ff679364e2f808bfd5f19b25fda P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:56.849298 16102 raft_consensus.cc:2781] T 4a6a5ff679364e2f808bfd5f19b25fda P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:56.849412 16102 raft_consensus.cc:686] T 4a6a5ff679364e2f808bfd5f19b25fda P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 LEADER]: Becoming Leader. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Running, Role: LEADER
I20240503 01:59:56.849567 16102 consensus_queue.cc:237] T 4a6a5ff679364e2f808bfd5f19b25fda P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:56.852941 6739 catalog_manager.cc:5484] T 4a6a5ff679364e2f808bfd5f19b25fda P b7062fbbc2b54fff9e99ec4ea5562c76 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:56.873476 9605 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:56.879724 7256 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:56.887701 7125 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 01:59:56.891021 6861 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:56.893458 9473 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:56.897238 6739 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:56.902266 9341 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 01:59:56.913362 6993 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 01:59:56.915802 16235 tablet.cc:1722] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:56.921742 16236 tablet.cc:1722] T 1443f81724f74fc99cf10616032f24ac P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:56.927816 6739 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=c0055f5131754db8beafabfa1f12d875] alter complete (version 3)
I20240503 01:59:56.932924 16239 tablet.cc:1722] T 1443f81724f74fc99cf10616032f24ac P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:56.945031 9735 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:57.007186 14833 consensus_queue.cc:1035] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:57.010421 14833 consensus_queue.cc:1035] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:57.030691 6734 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:57.030879 6734 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:57.033576 6734 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 1443f81724f74fc99cf10616032f24ac
I20240503 01:59:57.034121 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet 1443f81724f74fc99cf10616032f24ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:57 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:57.034436 16284 tablet_replica.cc:330] T 1443f81724f74fc99cf10616032f24ac P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:57.034583 9473 tablet_service.cc:1514] Processing DeleteTablet for tablet 1443f81724f74fc99cf10616032f24ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:57 UTC) from {username='slave'} at 127.0.0.1:54678
I20240503 01:59:57.034643 16284 raft_consensus.cc:2229] T 1443f81724f74fc99cf10616032f24ac P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:57.034780 16284 raft_consensus.cc:2258] T 1443f81724f74fc99cf10616032f24ac P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:57.035045 7125 tablet_service.cc:1514] Processing DeleteTablet for tablet 1443f81724f74fc99cf10616032f24ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:57 UTC) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:57.035357 16290 tablet_replica.cc:330] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:57.035509 16290 raft_consensus.cc:2229] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:57.035739 16290 raft_consensus.cc:2258] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:57.036207 16289 tablet_replica.cc:330] T 1443f81724f74fc99cf10616032f24ac P bb5364d730cb4489a0bfc6c270b7450a: stopping tablet replica
I20240503 01:59:57.036442 16289 raft_consensus.cc:2229] T 1443f81724f74fc99cf10616032f24ac P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:57.036466 16284 ts_tablet_manager.cc:1875] T 1443f81724f74fc99cf10616032f24ac P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:57.036475 16290 ts_tablet_manager.cc:1875] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:57.036690 16289 raft_consensus.cc:2258] T 1443f81724f74fc99cf10616032f24ac P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:57.037364 16289 ts_tablet_manager.cc:1875] T 1443f81724f74fc99cf10616032f24ac P bb5364d730cb4489a0bfc6c270b7450a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:57.039109 16290 ts_tablet_manager.cc:1888] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:57.039254 16290 log.cc:1196] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/1443f81724f74fc99cf10616032f24ac
I20240503 01:59:57.039551 16284 ts_tablet_manager.cc:1888] T 1443f81724f74fc99cf10616032f24ac P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:57.039665 16290 ts_tablet_manager.cc:1909] T 1443f81724f74fc99cf10616032f24ac P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting consensus metadata
I20240503 01:59:57.039923 16284 log.cc:1196] T 1443f81724f74fc99cf10616032f24ac P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/1443f81724f74fc99cf10616032f24ac
I20240503 01:59:57.040439 16284 ts_tablet_manager.cc:1909] T 1443f81724f74fc99cf10616032f24ac P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:57.040678 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet 1443f81724f74fc99cf10616032f24ac (table rebalancer_dynamic_table_new_name [id=c0055f5131754db8beafabfa1f12d875]) successfully deleted
I20240503 01:59:57.041139 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 1443f81724f74fc99cf10616032f24ac (table rebalancer_dynamic_table_new_name [id=c0055f5131754db8beafabfa1f12d875]) successfully deleted
I20240503 01:59:57.041218 16289 ts_tablet_manager.cc:1888] T 1443f81724f74fc99cf10616032f24ac P bb5364d730cb4489a0bfc6c270b7450a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:57.041299 16289 log.cc:1196] T 1443f81724f74fc99cf10616032f24ac P bb5364d730cb4489a0bfc6c270b7450a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/1443f81724f74fc99cf10616032f24ac
I20240503 01:59:57.041568 16289 ts_tablet_manager.cc:1909] T 1443f81724f74fc99cf10616032f24ac P bb5364d730cb4489a0bfc6c270b7450a: Deleting consensus metadata
I20240503 01:59:57.042448 6725 catalog_manager.cc:4830] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): tablet 1443f81724f74fc99cf10616032f24ac (table rebalancer_dynamic_table_new_name [id=c0055f5131754db8beafabfa1f12d875]) successfully deleted
I20240503 01:59:57.098953 16231 consensus_queue.cc:1035] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:57.112160 16231 consensus_queue.cc:1035] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:57.112267 6883 raft_consensus.cc:1203] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 01:59:57.136821 6737 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:57.157366 9473 tablet_service.cc:1467] Processing CreateTablet for tablet f85c9caa9753487b81c48feec8e85eb2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=66226533cc3e4c2bb66c02ac2645702b]), partition=
I20240503 01:59:57.157447 6861 tablet_service.cc:1467] Processing CreateTablet for tablet f85c9caa9753487b81c48feec8e85eb2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=66226533cc3e4c2bb66c02ac2645702b]), partition=
I20240503 01:59:57.157703 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f85c9caa9753487b81c48feec8e85eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:57.157737 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f85c9caa9753487b81c48feec8e85eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:57.160508 7256 tablet_service.cc:1467] Processing CreateTablet for tablet f85c9caa9753487b81c48feec8e85eb2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=66226533cc3e4c2bb66c02ac2645702b]), partition=
I20240503 01:59:57.160753 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f85c9caa9753487b81c48feec8e85eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:57.161761 7254 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:57.171389 16327 tablet_bootstrap.cc:492] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:57.172350 16326 tablet_bootstrap.cc:492] T f85c9caa9753487b81c48feec8e85eb2 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:57.173506 16326 tablet_bootstrap.cc:654] T f85c9caa9753487b81c48feec8e85eb2 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:57.174690 16327 tablet_bootstrap.cc:654] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:57.178913 6993 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 01:59:57.178809 6861 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:57.179338 16326 tablet_bootstrap.cc:492] T f85c9caa9753487b81c48feec8e85eb2 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:57.179579 16326 ts_tablet_manager.cc:1367] T f85c9caa9753487b81c48feec8e85eb2 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 01:59:57.182271 9605 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:57.183518 9341 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 01:59:57.184312 16326 raft_consensus.cc:348] T f85c9caa9753487b81c48feec8e85eb2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:57.184561 16326 raft_consensus.cc:374] T f85c9caa9753487b81c48feec8e85eb2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:57.184655 16326 raft_consensus.cc:729] T f85c9caa9753487b81c48feec8e85eb2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:57.184895 16326 consensus_queue.cc:260] T f85c9caa9753487b81c48feec8e85eb2 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:57.185433 16326 ts_tablet_manager.cc:1398] T f85c9caa9753487b81c48feec8e85eb2 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:57.186388 16327 tablet_bootstrap.cc:492] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:57.186616 16327 ts_tablet_manager.cc:1367] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20240503 01:59:57.187323 16327 raft_consensus.cc:348] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:57.187690 16327 raft_consensus.cc:374] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:57.187783 16327 raft_consensus.cc:729] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:57.187978 16327 consensus_queue.cc:260] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:57.189373 16328 tablet_bootstrap.cc:492] T f85c9caa9753487b81c48feec8e85eb2 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:57.189499 9473 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:57.190249 16327 ts_tablet_manager.cc:1398] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:57.200241 16328 tablet_bootstrap.cc:654] T f85c9caa9753487b81c48feec8e85eb2 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:57.217737 9735 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:57.223381 7125 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 01:59:57.226989 16328 tablet_bootstrap.cc:492] T f85c9caa9753487b81c48feec8e85eb2 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:57.227376 16328 ts_tablet_manager.cc:1367] T f85c9caa9753487b81c48feec8e85eb2 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
I20240503 01:59:57.229308 16328 raft_consensus.cc:348] T f85c9caa9753487b81c48feec8e85eb2 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:57.230330 16328 raft_consensus.cc:374] T f85c9caa9753487b81c48feec8e85eb2 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:57.230412 16328 raft_consensus.cc:729] T f85c9caa9753487b81c48feec8e85eb2 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:57.230676 16328 consensus_queue.cc:260] T f85c9caa9753487b81c48feec8e85eb2 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:57.231745 16328 ts_tablet_manager.cc:1398] T f85c9caa9753487b81c48feec8e85eb2 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:57.302325 7012 consensus_queue.cc:237] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:57.303886 9757 raft_consensus.cc:1261] T 487eafe47df84b468618f473eca37874 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:57.304159 9493 raft_consensus.cc:1261] T 487eafe47df84b468618f473eca37874 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:57.304405 16187 consensus_queue.cc:1035] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:57.305694 16187 consensus_queue.cc:1035] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:57.308200 16187 raft_consensus.cc:2932] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:57.311292 9493 raft_consensus.cc:2932] T 487eafe47df84b468618f473eca37874 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:57.312016 6737 catalog_manager.cc:5484] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:57.317873 9341 tablet_service.cc:1514] Processing DeleteTablet for tablet 487eafe47df84b468618f473eca37874 with delete_type TABLET_DATA_TOMBSTONED (TS 12e8c683db054df3b75c6285b98a5ad8 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:36040
I20240503 01:59:57.318264 16354 tablet_replica.cc:330] T 487eafe47df84b468618f473eca37874 P 12e8c683db054df3b75c6285b98a5ad8: stopping tablet replica
I20240503 01:59:57.318585 16354 raft_consensus.cc:2229] T 487eafe47df84b468618f473eca37874 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:57.318761 16004 raft_consensus.cc:482] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:57.318867 16004 raft_consensus.cc:504] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:57.318962 16354 raft_consensus.cc:2258] T 487eafe47df84b468618f473eca37874 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:57.319288 16004 leader_election.cc:288] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:57.319686 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85c9caa9753487b81c48feec8e85eb2" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:57.319718 6883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85c9caa9753487b81c48feec8e85eb2" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:57.319833 6883 raft_consensus.cc:2454] T f85c9caa9753487b81c48feec8e85eb2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:57.319834 7274 raft_consensus.cc:2454] T f85c9caa9753487b81c48feec8e85eb2 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:57.320120 9430 leader_election.cc:302] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:57.320370 16004 raft_consensus.cc:2781] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:57.320442 16004 raft_consensus.cc:482] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:57.320485 16004 raft_consensus.cc:3037] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:57.320462 16354 ts_tablet_manager.cc:1875] T 487eafe47df84b468618f473eca37874 P 12e8c683db054df3b75c6285b98a5ad8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:57.320885 9757 raft_consensus.cc:2932] T 487eafe47df84b468618f473eca37874 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:57.321617 16004 raft_consensus.cc:504] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:57.321998 16004 leader_election.cc:288] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Requested vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:57.322685 6883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85c9caa9753487b81c48feec8e85eb2" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:57.322824 6883 raft_consensus.cc:3037] T f85c9caa9753487b81c48feec8e85eb2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:57.323091 16354 ts_tablet_manager.cc:1888] T 487eafe47df84b468618f473eca37874 P 12e8c683db054df3b75c6285b98a5ad8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 01:59:57.323158 16354 log.cc:1196] T 487eafe47df84b468618f473eca37874 P 12e8c683db054df3b75c6285b98a5ad8: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/487eafe47df84b468618f473eca37874
I20240503 01:59:57.323876 6883 raft_consensus.cc:2454] T f85c9caa9753487b81c48feec8e85eb2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:57.324178 9430 leader_election.cc:302] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:57.324254 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f85c9caa9753487b81c48feec8e85eb2" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:57.324344 16004 raft_consensus.cc:2781] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:57.324412 16004 raft_consensus.cc:686] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Becoming Leader. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Running, Role: LEADER
I20240503 01:59:57.324398 7274 raft_consensus.cc:3037] T f85c9caa9753487b81c48feec8e85eb2 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:57.324558 16004 consensus_queue.cc:237] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:57.324710 6727 catalog_manager.cc:4830] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): tablet 487eafe47df84b468618f473eca37874 (table rebalancer_extra_table_12 [id=d46b94d9de0c43ad876391aee881e04d]) successfully deleted
I20240503 01:59:57.325716 7274 raft_consensus.cc:2454] T f85c9caa9753487b81c48feec8e85eb2 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:57.328210 6734 catalog_manager.cc:5484] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: term changed from 0 to 1, leader changed from <none> to bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 01:59:57.376454 16102 consensus_queue.cc:1035] T 4a6a5ff679364e2f808bfd5f19b25fda P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:57.381420 16102 consensus_queue.cc:1035] T 4a6a5ff679364e2f808bfd5f19b25fda P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:57.430219 6883 consensus_queue.cc:237] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:57.432744 7143 raft_consensus.cc:1261] T 2065dcd6cde84f9a928e4b817737dd42 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:57.432816 7012 raft_consensus.cc:1261] T 2065dcd6cde84f9a928e4b817737dd42 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:57.433233 16401 consensus_queue.cc:1035] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:57.433701 14781 consensus_queue.cc:1035] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 01:59:57.434017 6818 consensus_peers.cc:485] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2065dcd6cde84f9a928e4b817737dd42. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:57.437304 16401 raft_consensus.cc:2932] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:57.438338 7012 raft_consensus.cc:2932] T 2065dcd6cde84f9a928e4b817737dd42 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:57.440725 7143 raft_consensus.cc:2932] T 2065dcd6cde84f9a928e4b817737dd42 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:57.443845 6734 catalog_manager.cc:5484] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:57.450496 6732 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:57.456552 7274 raft_consensus.cc:1261] T f85c9caa9753487b81c48feec8e85eb2 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:57.456746 6883 raft_consensus.cc:1261] T f85c9caa9753487b81c48feec8e85eb2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:57.457295 16004 consensus_queue.cc:1035] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:57.457670 16004 consensus_queue.cc:1035] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:57.460860 6737 catalog_manager.cc:5576] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271) does not have the latest schema for tablet f85c9caa9753487b81c48feec8e85eb2 (table rebalancer_dynamic_table [id=66226533cc3e4c2bb66c02ac2645702b]). Expected version 1 got 0
I20240503 01:59:57.461751 6732 catalog_manager.cc:5576] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943) does not have the latest schema for tablet f85c9caa9753487b81c48feec8e85eb2 (table rebalancer_dynamic_table [id=66226533cc3e4c2bb66c02ac2645702b]). Expected version 1 got 0
I20240503 01:59:57.462965 6737 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:57.464753 16418 tablet.cc:1722] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:57.465224 16418 tablet.cc:1620] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:57.465562 6732 catalog_manager.cc:5576] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943) does not have the latest schema for tablet f85c9caa9753487b81c48feec8e85eb2 (table rebalancer_dynamic_table [id=66226533cc3e4c2bb66c02ac2645702b]). Expected version 1 got 0
I20240503 01:59:57.468611 6740 catalog_manager.cc:5576] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271) does not have the latest schema for tablet f85c9caa9753487b81c48feec8e85eb2 (table rebalancer_dynamic_table [id=66226533cc3e4c2bb66c02ac2645702b]). Expected version 1 got 0
I20240503 01:59:57.472266 16419 tablet.cc:1722] T f85c9caa9753487b81c48feec8e85eb2 P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:57.472640 16419 tablet.cc:1620] T f85c9caa9753487b81c48feec8e85eb2 P 5dcc33cc621a4cbbae155f3230fb7c46: MemRowSet was empty: no flush needed.
I20240503 01:59:57.474507 9735 tablet_service.cc:1467] Processing CreateTablet for tablet c4930b3ca84749f6a4cd245411f66075 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=6f17b2a98eb44079957e3e2ae659fb2c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:57.474622 7125 tablet_service.cc:1467] Processing CreateTablet for tablet c4930b3ca84749f6a4cd245411f66075 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=6f17b2a98eb44079957e3e2ae659fb2c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:57.474874 9735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4930b3ca84749f6a4cd245411f66075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:57.474936 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4930b3ca84749f6a4cd245411f66075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:57.475234 16420 tablet.cc:1722] T f85c9caa9753487b81c48feec8e85eb2 P 1d0e738c34af40fa9950c6f086619b3a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:57.475567 16420 tablet.cc:1620] T f85c9caa9753487b81c48feec8e85eb2 P 1d0e738c34af40fa9950c6f086619b3a: MemRowSet was empty: no flush needed.
I20240503 01:59:57.477494 16421 tablet_bootstrap.cc:492] T c4930b3ca84749f6a4cd245411f66075 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:57.478652 6993 tablet_service.cc:1467] Processing CreateTablet for tablet 5e20b869a89e44f1b0ba854163c31f06 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=6f17b2a98eb44079957e3e2ae659fb2c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:57.478929 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e20b869a89e44f1b0ba854163c31f06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:57.478878 7124 tablet_service.cc:1467] Processing CreateTablet for tablet 5e20b869a89e44f1b0ba854163c31f06 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=6f17b2a98eb44079957e3e2ae659fb2c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:57.479094 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e20b869a89e44f1b0ba854163c31f06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:57.479090 9341 tablet_service.cc:1467] Processing CreateTablet for tablet c4930b3ca84749f6a4cd245411f66075 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=6f17b2a98eb44079957e3e2ae659fb2c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:57.479362 9341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4930b3ca84749f6a4cd245411f66075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:57.481310 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 5e20b869a89e44f1b0ba854163c31f06 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=6f17b2a98eb44079957e3e2ae659fb2c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:57.481484 16421 tablet_bootstrap.cc:654] T c4930b3ca84749f6a4cd245411f66075 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:57.481678 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e20b869a89e44f1b0ba854163c31f06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:57.482327 16423 tablet_bootstrap.cc:492] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:57.482329 9735 tablet_service.cc:1467] Processing CreateTablet for tablet 75175ed25db846b9ba8e8e7b595102ca (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=6f17b2a98eb44079957e3e2ae659fb2c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:57.482880 9735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75175ed25db846b9ba8e8e7b595102ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:57.483361 16424 tablet_bootstrap.cc:492] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:57.483584 16423 tablet_bootstrap.cc:654] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:57.484766 16423 tablet_bootstrap.cc:492] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:57.484864 16423 ts_tablet_manager.cc:1367] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:57.485327 16423 raft_consensus.cc:348] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:57.485443 16423 raft_consensus.cc:374] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:57.485478 16423 raft_consensus.cc:729] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:57.485605 16423 consensus_queue.cc:260] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:57.485893 16423 ts_tablet_manager.cc:1398] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:57.486044 9473 tablet_service.cc:1467] Processing CreateTablet for tablet 75175ed25db846b9ba8e8e7b595102ca (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=6f17b2a98eb44079957e3e2ae659fb2c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:57.486320 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75175ed25db846b9ba8e8e7b595102ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:57.488040 7125 tablet_service.cc:1467] Processing CreateTablet for tablet 75175ed25db846b9ba8e8e7b595102ca (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=6f17b2a98eb44079957e3e2ae659fb2c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:57.488368 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75175ed25db846b9ba8e8e7b595102ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:57.488569 16327 tablet_bootstrap.cc:492] T 75175ed25db846b9ba8e8e7b595102ca P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:57.489091 6740 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=66226533cc3e4c2bb66c02ac2645702b] alter complete (version 1)
I20240503 01:59:57.489547 16327 tablet_bootstrap.cc:654] T 75175ed25db846b9ba8e8e7b595102ca P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:57.490566 16422 tablet_bootstrap.cc:492] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:57.490904 16424 tablet_bootstrap.cc:654] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:57.491797 16422 tablet_bootstrap.cc:654] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:57.493979 16424 tablet_bootstrap.cc:492] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:57.494096 16424 ts_tablet_manager.cc:1367] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 01:59:57.494632 16424 raft_consensus.cc:348] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:57.495136 16422 tablet_bootstrap.cc:492] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:57.495222 16422 ts_tablet_manager.cc:1367] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 01:59:57.495342 16424 raft_consensus.cc:374] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:57.495388 16424 raft_consensus.cc:729] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:57.495535 16424 consensus_queue.cc:260] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:57.495733 16422 raft_consensus.cc:348] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:57.495828 16422 raft_consensus.cc:374] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:57.495858 16422 raft_consensus.cc:729] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:57.495970 16422 consensus_queue.cc:260] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:57.496263 16422 ts_tablet_manager.cc:1398] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:57.496424 16422 tablet_bootstrap.cc:492] T 75175ed25db846b9ba8e8e7b595102ca P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:57.497431 16422 tablet_bootstrap.cc:654] T 75175ed25db846b9ba8e8e7b595102ca P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:57.499011 16424 ts_tablet_manager.cc:1398] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:57.499783 16327 tablet_bootstrap.cc:492] T 75175ed25db846b9ba8e8e7b595102ca P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:57.499871 16327 ts_tablet_manager.cc:1367] T 75175ed25db846b9ba8e8e7b595102ca P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 01:59:57.500447 16327 raft_consensus.cc:348] T 75175ed25db846b9ba8e8e7b595102ca P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:57.500568 16327 raft_consensus.cc:374] T 75175ed25db846b9ba8e8e7b595102ca P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:57.500597 16327 raft_consensus.cc:729] T 75175ed25db846b9ba8e8e7b595102ca P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:57.500726 16327 consensus_queue.cc:260] T 75175ed25db846b9ba8e8e7b595102ca P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:57.500861 16424 tablet_bootstrap.cc:492] T 5e20b869a89e44f1b0ba854163c31f06 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:57.500962 16327 ts_tablet_manager.cc:1398] T 75175ed25db846b9ba8e8e7b595102ca P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:57.502077 16424 tablet_bootstrap.cc:654] T 5e20b869a89e44f1b0ba854163c31f06 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:57.507310 16421 tablet_bootstrap.cc:492] T c4930b3ca84749f6a4cd245411f66075 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:57.507442 16421 ts_tablet_manager.cc:1367] T c4930b3ca84749f6a4cd245411f66075 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.000s
I20240503 01:59:57.508698 16421 raft_consensus.cc:348] T c4930b3ca84749f6a4cd245411f66075 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:57.508831 16421 raft_consensus.cc:374] T c4930b3ca84749f6a4cd245411f66075 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:57.508865 16421 raft_consensus.cc:729] T c4930b3ca84749f6a4cd245411f66075 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:57.509002 16421 consensus_queue.cc:260] T c4930b3ca84749f6a4cd245411f66075 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:57.509421 16421 ts_tablet_manager.cc:1398] T c4930b3ca84749f6a4cd245411f66075 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:57.509712 16421 tablet_bootstrap.cc:492] T 5e20b869a89e44f1b0ba854163c31f06 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:57.510818 16421 tablet_bootstrap.cc:654] T 5e20b869a89e44f1b0ba854163c31f06 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:57.510954 16422 tablet_bootstrap.cc:492] T 75175ed25db846b9ba8e8e7b595102ca P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:57.511046 16422 ts_tablet_manager.cc:1367] T 75175ed25db846b9ba8e8e7b595102ca P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 01:59:57.511611 16422 raft_consensus.cc:348] T 75175ed25db846b9ba8e8e7b595102ca P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:57.511739 16422 raft_consensus.cc:374] T 75175ed25db846b9ba8e8e7b595102ca P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:57.511773 16422 raft_consensus.cc:729] T 75175ed25db846b9ba8e8e7b595102ca P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:57.513202 16422 consensus_queue.cc:260] T 75175ed25db846b9ba8e8e7b595102ca P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:57.515312 16421 tablet_bootstrap.cc:492] T 5e20b869a89e44f1b0ba854163c31f06 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:57.515422 16421 ts_tablet_manager.cc:1367] T 5e20b869a89e44f1b0ba854163c31f06 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 01:59:57.515586 16422 ts_tablet_manager.cc:1398] T 75175ed25db846b9ba8e8e7b595102ca P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 01:59:57.515918 16421 raft_consensus.cc:348] T 5e20b869a89e44f1b0ba854163c31f06 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:57.516265 16421 raft_consensus.cc:374] T 5e20b869a89e44f1b0ba854163c31f06 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:57.516310 16421 raft_consensus.cc:729] T 5e20b869a89e44f1b0ba854163c31f06 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:57.516454 16421 consensus_queue.cc:260] T 5e20b869a89e44f1b0ba854163c31f06 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:57.516685 16424 tablet_bootstrap.cc:492] T 5e20b869a89e44f1b0ba854163c31f06 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:57.516748 16421 ts_tablet_manager.cc:1398] T 5e20b869a89e44f1b0ba854163c31f06 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:57.516799 16424 ts_tablet_manager.cc:1367] T 5e20b869a89e44f1b0ba854163c31f06 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 01:59:57.517153 16421 tablet_bootstrap.cc:492] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:57.517297 16424 raft_consensus.cc:348] T 5e20b869a89e44f1b0ba854163c31f06 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:57.517419 16424 raft_consensus.cc:374] T 5e20b869a89e44f1b0ba854163c31f06 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:57.517457 16424 raft_consensus.cc:729] T 5e20b869a89e44f1b0ba854163c31f06 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:57.517619 16424 consensus_queue.cc:260] T 5e20b869a89e44f1b0ba854163c31f06 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:57.517917 16424 ts_tablet_manager.cc:1398] T 5e20b869a89e44f1b0ba854163c31f06 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:57.518354 16421 tablet_bootstrap.cc:654] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:57.523342 16421 tablet_bootstrap.cc:492] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:57.523525 16421 ts_tablet_manager.cc:1367] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 01:59:57.524027 16421 raft_consensus.cc:348] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:57.524214 16421 raft_consensus.cc:374] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:57.524283 16421 raft_consensus.cc:729] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:57.524449 16421 consensus_queue.cc:260] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:57.525080 16421 ts_tablet_manager.cc:1398] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:57.533552 6883 consensus_queue.cc:237] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:57.535902 7143 raft_consensus.cc:1261] T 4bd8226227c348d5a5d4d8ff28959d27 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:57.536288 7276 raft_consensus.cc:1261] T 4bd8226227c348d5a5d4d8ff28959d27 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:57.536378 6818 consensus_peers.cc:485] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4bd8226227c348d5a5d4d8ff28959d27. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:57.536701 16401 consensus_queue.cc:1035] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:57.536924 14781 consensus_queue.cc:1035] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:57.539865 14781 raft_consensus.cc:2932] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:57.542685 7276 raft_consensus.cc:2932] T 4bd8226227c348d5a5d4d8ff28959d27 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:57.543815 6732 catalog_manager.cc:5484] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:57.544628 7143 raft_consensus.cc:2932] T 4bd8226227c348d5a5d4d8ff28959d27 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:57.553707 15159 raft_consensus.cc:482] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:57.553822 15159 raft_consensus.cc:504] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:57.554275 15159 leader_election.cc:288] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:57.554878 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75175ed25db846b9ba8e8e7b595102ca" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:57.555014 9493 raft_consensus.cc:2454] T 75175ed25db846b9ba8e8e7b595102ca P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:57.555310 7078 leader_election.cc:302] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:57.555392 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75175ed25db846b9ba8e8e7b595102ca" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:57.555521 9757 raft_consensus.cc:2454] T 75175ed25db846b9ba8e8e7b595102ca P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:57.555605 15957 raft_consensus.cc:2781] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:57.555651 15957 raft_consensus.cc:482] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:57.555684 15957 raft_consensus.cc:3037] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:57.556807 15957 raft_consensus.cc:504] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:57.557145 15957 leader_election.cc:288] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Requested vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:57.557548 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75175ed25db846b9ba8e8e7b595102ca" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:57.557675 9493 raft_consensus.cc:3037] T 75175ed25db846b9ba8e8e7b595102ca P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:57.557627 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75175ed25db846b9ba8e8e7b595102ca" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:57.557785 9757 raft_consensus.cc:3037] T 75175ed25db846b9ba8e8e7b595102ca P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:57.558920 9757 raft_consensus.cc:2454] T 75175ed25db846b9ba8e8e7b595102ca P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:57.559427 9493 raft_consensus.cc:2454] T 75175ed25db846b9ba8e8e7b595102ca P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:57.560108 7078 leader_election.cc:302] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:57.560503 15159 raft_consensus.cc:2781] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:57.560588 15159 raft_consensus.cc:686] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Becoming Leader. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Running, Role: LEADER
I20240503 01:59:57.560748 15159 consensus_queue.cc:237] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:57.564402 6738 catalog_manager.cc:5484] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:57.584378 6883 consensus_queue.cc:237] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:57.585930 7012 raft_consensus.cc:1261] T c4ac4ad7f9eb4723814f65ce96758c0e P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 01:59:57.586079 6818 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c4ac4ad7f9eb4723814f65ce96758c0e. This is attempt 1: this message will repeat every 5th retry.
I20240503 01:59:57.586391 15668 consensus_queue.cc:1035] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:57.586557 7143 raft_consensus.cc:1261] T c4ac4ad7f9eb4723814f65ce96758c0e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 01:59:57.587034 15668 consensus_queue.cc:1035] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 01:59:57.589438 16442 raft_consensus.cc:2932] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:57.590054 7143 raft_consensus.cc:2932] T c4ac4ad7f9eb4723814f65ce96758c0e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:57.592965 7012 raft_consensus.cc:2932] T c4ac4ad7f9eb4723814f65ce96758c0e P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:57.594022 6737 catalog_manager.cc:5484] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index -1 to 4, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:57.594931 15247 raft_consensus.cc:482] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:57.595031 15247 raft_consensus.cc:504] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:57.595477 15247 leader_election.cc:288] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:57.595441 6738 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:57.599874 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4930b3ca84749f6a4cd245411f66075" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:57.600062 7145 raft_consensus.cc:2454] T c4930b3ca84749f6a4cd245411f66075 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:57.601052 9298 leader_election.cc:302] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:57.601984 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4930b3ca84749f6a4cd245411f66075" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:57.602130 9757 raft_consensus.cc:2454] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 01:59:57.602833 15247 raft_consensus.cc:2781] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:57.602916 15247 raft_consensus.cc:482] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:57.602962 15247 raft_consensus.cc:3037] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:57.604601 15247 raft_consensus.cc:504] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:57.604979 15247 leader_election.cc:288] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Requested vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:57.605906 16418 tablet.cc:1722] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:57.606079 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4930b3ca84749f6a4cd245411f66075" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:57.606078 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4930b3ca84749f6a4cd245411f66075" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:57.606202 9757 raft_consensus.cc:3037] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:57.606251 7145 raft_consensus.cc:3037] T c4930b3ca84749f6a4cd245411f66075 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:57.606284 16418 tablet.cc:1620] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:57.607177 16420 tablet.cc:1722] T f85c9caa9753487b81c48feec8e85eb2 P 1d0e738c34af40fa9950c6f086619b3a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:57.607406 9757 raft_consensus.cc:2454] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:57.607405 7145 raft_consensus.cc:2454] T c4930b3ca84749f6a4cd245411f66075 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 01:59:57.607527 16420 tablet.cc:1620] T f85c9caa9753487b81c48feec8e85eb2 P 1d0e738c34af40fa9950c6f086619b3a: MemRowSet was empty: no flush needed.
I20240503 01:59:57.607921 9298 leader_election.cc:302] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:57.608400 15247 raft_consensus.cc:2781] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:57.608492 15247 raft_consensus.cc:686] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Running, Role: LEADER
I20240503 01:59:57.608635 15247 consensus_queue.cc:237] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:57.611224 6732 catalog_manager.cc:5484] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:57.613752 16419 tablet.cc:1722] T f85c9caa9753487b81c48feec8e85eb2 P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:57.614100 16419 tablet.cc:1620] T f85c9caa9753487b81c48feec8e85eb2 P 5dcc33cc621a4cbbae155f3230fb7c46: MemRowSet was empty: no flush needed.
I20240503 01:59:57.618233 6740 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=66226533cc3e4c2bb66c02ac2645702b] alter complete (version 2)
I20240503 01:59:57.720594 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:57.727409 16418 tablet.cc:1722] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:57.727916 16419 tablet.cc:1722] T f85c9caa9753487b81c48feec8e85eb2 P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:57.731142 16420 tablet.cc:1722] T f85c9caa9753487b81c48feec8e85eb2 P 1d0e738c34af40fa9950c6f086619b3a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:57.734618 6739 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=66226533cc3e4c2bb66c02ac2645702b] alter complete (version 3)
I20240503 01:59:57.784475 14833 raft_consensus.cc:482] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:57.784636 14833 raft_consensus.cc:504] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:57.785079 14833 leader_election.cc:288] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:57.785383 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e20b869a89e44f1b0ba854163c31f06" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:57.785537 7145 raft_consensus.cc:2454] T 5e20b869a89e44f1b0ba854163c31f06 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:57.785554 9361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e20b869a89e44f1b0ba854163c31f06" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:57.785702 9361 raft_consensus.cc:2454] T 5e20b869a89e44f1b0ba854163c31f06 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:57.785817 6949 leader_election.cc:302] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:57.785982 14833 raft_consensus.cc:2781] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:57.786036 14833 raft_consensus.cc:482] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:57.786095 14833 raft_consensus.cc:3037] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:57.787127 14833 raft_consensus.cc:504] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:57.787518 14833 leader_election.cc:288] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 01:59:57.788215 9361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e20b869a89e44f1b0ba854163c31f06" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:57.788344 9361 raft_consensus.cc:3037] T 5e20b869a89e44f1b0ba854163c31f06 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:57.788218 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e20b869a89e44f1b0ba854163c31f06" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:57.788487 7145 raft_consensus.cc:3037] T 5e20b869a89e44f1b0ba854163c31f06 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:57.789297 9361 raft_consensus.cc:2454] T 5e20b869a89e44f1b0ba854163c31f06 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:57.789361 7145 raft_consensus.cc:2454] T 5e20b869a89e44f1b0ba854163c31f06 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:57.789842 6949 leader_election.cc:302] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:57.790293 14833 raft_consensus.cc:2781] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:57.790450 14833 raft_consensus.cc:686] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:57.790627 14833 consensus_queue.cc:237] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:57.793169 6734 catalog_manager.cc:5484] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:57.823164 6861 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:57.827170 9342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 01:59:57.837706 6734 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:57.837896 6734 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:57.839560 9473 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:57.841723 6734 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet f85c9caa9753487b81c48feec8e85eb2
I20240503 01:59:57.842339 9735 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:57.842885 7256 tablet_service.cc:1514] Processing DeleteTablet for tablet f85c9caa9753487b81c48feec8e85eb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:57 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:57.843153 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet f85c9caa9753487b81c48feec8e85eb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:57 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:57.843233 16523 tablet_replica.cc:330] T f85c9caa9753487b81c48feec8e85eb2 P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:57.843389 16523 raft_consensus.cc:2229] T f85c9caa9753487b81c48feec8e85eb2 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:57.843518 16523 raft_consensus.cc:2258] T f85c9caa9753487b81c48feec8e85eb2 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:57.843789 9473 tablet_service.cc:1514] Processing DeleteTablet for tablet f85c9caa9753487b81c48feec8e85eb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:57 UTC) from {username='slave'} at 127.0.0.1:54678
I20240503 01:59:57.843924 16524 tablet_replica.cc:330] T f85c9caa9753487b81c48feec8e85eb2 P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:57.844079 16524 raft_consensus.cc:2229] T f85c9caa9753487b81c48feec8e85eb2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:57.844195 16524 raft_consensus.cc:2258] T f85c9caa9753487b81c48feec8e85eb2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:57.845487 16523 ts_tablet_manager.cc:1875] T f85c9caa9753487b81c48feec8e85eb2 P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:57.848240 7256 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:57.851562 9605 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:57.854681 6993 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 01:59:57.869442 7125 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 01:59:57.870702 16525 tablet_replica.cc:330] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a: stopping tablet replica
I20240503 01:59:57.875442 16525 raft_consensus.cc:2229] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:57.876088 16525 raft_consensus.cc:2258] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:57.880720 16526 ts_tablet_manager.cc:902] T 2065dcd6cde84f9a928e4b817737dd42 P 12e8c683db054df3b75c6285b98a5ad8: Initiating tablet copy from peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:57.882712 16526 tablet_copy_client.cc:308] T 2065dcd6cde84f9a928e4b817737dd42 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:42943
I20240503 01:59:57.887169 6903 tablet_copy_service.cc:140] P 5dcc33cc621a4cbbae155f3230fb7c46: Received BeginTabletCopySession request for tablet 2065dcd6cde84f9a928e4b817737dd42 from peer 12e8c683db054df3b75c6285b98a5ad8 ({username='slave'} at 127.0.99.133:55147)
I20240503 01:59:57.887320 6903 tablet_copy_service.cc:161] P 5dcc33cc621a4cbbae155f3230fb7c46: Beginning new tablet copy session on tablet 2065dcd6cde84f9a928e4b817737dd42 from peer 12e8c683db054df3b75c6285b98a5ad8 at {username='slave'} at 127.0.99.133:55147: session id = 12e8c683db054df3b75c6285b98a5ad8-2065dcd6cde84f9a928e4b817737dd42
I20240503 01:59:57.888397 6903 tablet_copy_source_session.cc:205] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:57.890432 16523 ts_tablet_manager.cc:1888] T f85c9caa9753487b81c48feec8e85eb2 P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:57.848647 16524 ts_tablet_manager.cc:1875] T f85c9caa9753487b81c48feec8e85eb2 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:57.893000 16523 log.cc:1196] T f85c9caa9753487b81c48feec8e85eb2 P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/f85c9caa9753487b81c48feec8e85eb2
I20240503 01:59:57.893579 16523 ts_tablet_manager.cc:1909] T f85c9caa9753487b81c48feec8e85eb2 P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:57.895561 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet f85c9caa9753487b81c48feec8e85eb2 (table rebalancer_dynamic_table_new_name [id=66226533cc3e4c2bb66c02ac2645702b]) successfully deleted
I20240503 01:59:57.899694 16524 ts_tablet_manager.cc:1888] T f85c9caa9753487b81c48feec8e85eb2 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:57.899966 16524 log.cc:1196] T f85c9caa9753487b81c48feec8e85eb2 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/f85c9caa9753487b81c48feec8e85eb2
I20240503 01:59:57.903029 16526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2065dcd6cde84f9a928e4b817737dd42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:57.905077 16524 ts_tablet_manager.cc:1909] T f85c9caa9753487b81c48feec8e85eb2 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:57.905973 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet f85c9caa9753487b81c48feec8e85eb2 (table rebalancer_dynamic_table_new_name [id=66226533cc3e4c2bb66c02ac2645702b]) successfully deleted
I20240503 01:59:57.908914 16526 tablet_copy_client.cc:783] T 2065dcd6cde84f9a928e4b817737dd42 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:57.909133 16526 tablet_copy_client.cc:647] T 2065dcd6cde84f9a928e4b817737dd42 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:57.910349 16526 tablet_copy_client.cc:523] T 2065dcd6cde84f9a928e4b817737dd42 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:57.911830 16526 tablet_bootstrap.cc:492] T 2065dcd6cde84f9a928e4b817737dd42 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:57.917189 16525 ts_tablet_manager.cc:1875] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:57.919620 16525 ts_tablet_manager.cc:1888] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:57.919716 16525 log.cc:1196] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/f85c9caa9753487b81c48feec8e85eb2
I20240503 01:59:57.919983 16525 ts_tablet_manager.cc:1909] T f85c9caa9753487b81c48feec8e85eb2 P bb5364d730cb4489a0bfc6c270b7450a: Deleting consensus metadata
I20240503 01:59:57.920987 6725 catalog_manager.cc:4830] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): tablet f85c9caa9753487b81c48feec8e85eb2 (table rebalancer_dynamic_table_new_name [id=66226533cc3e4c2bb66c02ac2645702b]) successfully deleted
I20240503 01:59:57.934841 16526 tablet_bootstrap.cc:492] T 2065dcd6cde84f9a928e4b817737dd42 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:57.935205 16526 tablet_bootstrap.cc:492] T 2065dcd6cde84f9a928e4b817737dd42 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap complete.
I20240503 01:59:57.935518 16526 ts_tablet_manager.cc:1367] T 2065dcd6cde84f9a928e4b817737dd42 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.024s user 0.023s sys 0.001s
I20240503 01:59:57.936329 16526 raft_consensus.cc:348] T 2065dcd6cde84f9a928e4b817737dd42 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:57.936470 16526 raft_consensus.cc:729] T 2065dcd6cde84f9a928e4b817737dd42 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:57.936625 16526 consensus_queue.cc:260] T 2065dcd6cde84f9a928e4b817737dd42 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:57.936939 16526 ts_tablet_manager.cc:1398] T 2065dcd6cde84f9a928e4b817737dd42 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:57.937469 6903 tablet_copy_service.cc:342] P 5dcc33cc621a4cbbae155f3230fb7c46: Request end of tablet copy session 12e8c683db054df3b75c6285b98a5ad8-2065dcd6cde84f9a928e4b817737dd42 received from {username='slave'} at 127.0.99.133:55147
I20240503 01:59:57.937572 6903 tablet_copy_service.cc:433] P 5dcc33cc621a4cbbae155f3230fb7c46: ending tablet copy session 12e8c683db054df3b75c6285b98a5ad8-2065dcd6cde84f9a928e4b817737dd42 on tablet 2065dcd6cde84f9a928e4b817737dd42 with peer 12e8c683db054df3b75c6285b98a5ad8
I20240503 01:59:57.946754 6734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:57.958644 6861 tablet_service.cc:1467] Processing CreateTablet for tablet cef0e3380c4f47079780872d1515e77e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=63cb96ff9c214c618cca4cfade4c8614]), partition=
I20240503 01:59:57.959141 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef0e3380c4f47079780872d1515e77e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:57.959196 6993 tablet_service.cc:1467] Processing CreateTablet for tablet cef0e3380c4f47079780872d1515e77e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=63cb96ff9c214c618cca4cfade4c8614]), partition=
I20240503 01:59:57.959519 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef0e3380c4f47079780872d1515e77e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:57.958643 7256 tablet_service.cc:1467] Processing CreateTablet for tablet cef0e3380c4f47079780872d1515e77e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=63cb96ff9c214c618cca4cfade4c8614]), partition=
I20240503 01:59:57.960222 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef0e3380c4f47079780872d1515e77e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:57.962620 16423 tablet_bootstrap.cc:492] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:57.963313 16543 tablet_bootstrap.cc:492] T cef0e3380c4f47079780872d1515e77e P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:57.963722 16423 tablet_bootstrap.cc:654] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:57.964532 16543 tablet_bootstrap.cc:654] T cef0e3380c4f47079780872d1515e77e P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:57.965597 16423 tablet_bootstrap.cc:492] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:57.965513 15159 consensus_queue.cc:1035] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:57.965690 16423 ts_tablet_manager.cc:1367] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:57.966221 16423 raft_consensus.cc:348] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:57.966378 16423 raft_consensus.cc:374] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:57.966418 16423 raft_consensus.cc:729] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:57.966578 16423 consensus_queue.cc:260] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:57.966686 16543 tablet_bootstrap.cc:492] T cef0e3380c4f47079780872d1515e77e P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:57.966791 16543 ts_tablet_manager.cc:1367] T cef0e3380c4f47079780872d1515e77e P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 01:59:57.966904 16423 ts_tablet_manager.cc:1398] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:57.968032 16543 raft_consensus.cc:348] T cef0e3380c4f47079780872d1515e77e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:57.968235 16543 raft_consensus.cc:374] T cef0e3380c4f47079780872d1515e77e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:57.968274 16543 raft_consensus.cc:729] T cef0e3380c4f47079780872d1515e77e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:57.968546 16543 consensus_queue.cc:260] T cef0e3380c4f47079780872d1515e77e P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:57.969159 16543 ts_tablet_manager.cc:1398] T cef0e3380c4f47079780872d1515e77e P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:57.971477 15159 consensus_queue.cc:1035] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:57.972162 16544 tablet_bootstrap.cc:492] T cef0e3380c4f47079780872d1515e77e P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:57.973598 16544 tablet_bootstrap.cc:654] T cef0e3380c4f47079780872d1515e77e P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:57.975129 16544 tablet_bootstrap.cc:492] T cef0e3380c4f47079780872d1515e77e P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:57.975333 16544 ts_tablet_manager.cc:1367] T cef0e3380c4f47079780872d1515e77e P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 01:59:57.975792 16544 raft_consensus.cc:348] T cef0e3380c4f47079780872d1515e77e P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:57.975921 16544 raft_consensus.cc:374] T cef0e3380c4f47079780872d1515e77e P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:57.975950 16544 raft_consensus.cc:729] T cef0e3380c4f47079780872d1515e77e P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:57.976079 16544 consensus_queue.cc:260] T cef0e3380c4f47079780872d1515e77e P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:57.976446 16544 ts_tablet_manager.cc:1398] T cef0e3380c4f47079780872d1515e77e P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:58.040019 15247 consensus_queue.cc:1035] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:58.047015 15917 consensus_queue.cc:1035] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:58.047149 14833 raft_consensus.cc:482] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:58.047277 14833 raft_consensus.cc:504] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:58.048033 14833 leader_election.cc:288] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:58.048993 6883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef0e3380c4f47079780872d1515e77e" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:58.049203 6883 raft_consensus.cc:2454] T cef0e3380c4f47079780872d1515e77e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:58.049968 6950 leader_election.cc:302] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:58.050832 7276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef0e3380c4f47079780872d1515e77e" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:58.051023 7276 raft_consensus.cc:2454] T cef0e3380c4f47079780872d1515e77e P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:58.053918 14833 raft_consensus.cc:2781] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:58.054023 14833 raft_consensus.cc:482] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:58.054064 14833 raft_consensus.cc:3037] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:58.055331 14833 raft_consensus.cc:504] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:58.055754 14833 leader_election.cc:288] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:58.057243 6883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef0e3380c4f47079780872d1515e77e" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:58.057396 6883 raft_consensus.cc:3037] T cef0e3380c4f47079780872d1515e77e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:58.059019 6883 raft_consensus.cc:2454] T cef0e3380c4f47079780872d1515e77e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:58.059648 6950 leader_election.cc:302] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:58.059907 14833 raft_consensus.cc:2781] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:58.061300 14833 raft_consensus.cc:686] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:58.061488 14833 consensus_queue.cc:237] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:58.063879 7276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef0e3380c4f47079780872d1515e77e" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:58.064113 7276 raft_consensus.cc:3037] T cef0e3380c4f47079780872d1515e77e P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:58.064234 6741 catalog_manager.cc:5484] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:58.065753 7276 raft_consensus.cc:2454] T cef0e3380c4f47079780872d1515e77e P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:58.079241 16526 ts_tablet_manager.cc:902] T c4ac4ad7f9eb4723814f65ce96758c0e P 12e8c683db054df3b75c6285b98a5ad8: Initiating tablet copy from peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:58.079780 16526 tablet_copy_client.cc:308] T c4ac4ad7f9eb4723814f65ce96758c0e P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:42943
I20240503 01:59:58.080251 6903 tablet_copy_service.cc:140] P 5dcc33cc621a4cbbae155f3230fb7c46: Received BeginTabletCopySession request for tablet c4ac4ad7f9eb4723814f65ce96758c0e from peer 12e8c683db054df3b75c6285b98a5ad8 ({username='slave'} at 127.0.99.133:55147)
I20240503 01:59:58.080356 6903 tablet_copy_service.cc:161] P 5dcc33cc621a4cbbae155f3230fb7c46: Beginning new tablet copy session on tablet c4ac4ad7f9eb4723814f65ce96758c0e from peer 12e8c683db054df3b75c6285b98a5ad8 at {username='slave'} at 127.0.99.133:55147: session id = 12e8c683db054df3b75c6285b98a5ad8-c4ac4ad7f9eb4723814f65ce96758c0e
I20240503 01:59:58.081341 6903 tablet_copy_source_session.cc:205] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:58.082419 16526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ac4ad7f9eb4723814f65ce96758c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:58.085304 16526 tablet_copy_client.cc:783] T c4ac4ad7f9eb4723814f65ce96758c0e P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:58.085525 16526 tablet_copy_client.cc:647] T c4ac4ad7f9eb4723814f65ce96758c0e P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:58.097745 16590 ts_tablet_manager.cc:902] T 4bd8226227c348d5a5d4d8ff28959d27 P b7062fbbc2b54fff9e99ec4ea5562c76: Initiating tablet copy from peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:58.098500 16590 tablet_copy_client.cc:308] T 4bd8226227c348d5a5d4d8ff28959d27 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:42943
I20240503 01:59:58.099087 6903 tablet_copy_service.cc:140] P 5dcc33cc621a4cbbae155f3230fb7c46: Received BeginTabletCopySession request for tablet 4bd8226227c348d5a5d4d8ff28959d27 from peer b7062fbbc2b54fff9e99ec4ea5562c76 ({username='slave'} at 127.0.99.135:54599)
I20240503 01:59:58.099337 6903 tablet_copy_service.cc:161] P 5dcc33cc621a4cbbae155f3230fb7c46: Beginning new tablet copy session on tablet 4bd8226227c348d5a5d4d8ff28959d27 from peer b7062fbbc2b54fff9e99ec4ea5562c76 at {username='slave'} at 127.0.99.135:54599: session id = b7062fbbc2b54fff9e99ec4ea5562c76-4bd8226227c348d5a5d4d8ff28959d27
I20240503 01:59:58.100558 6903 tablet_copy_source_session.cc:205] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 01:59:58.102026 16590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bd8226227c348d5a5d4d8ff28959d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:58.102548 16526 tablet_copy_client.cc:523] T c4ac4ad7f9eb4723814f65ce96758c0e P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:58.104089 16526 tablet_bootstrap.cc:492] T c4ac4ad7f9eb4723814f65ce96758c0e P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:58.105197 16590 tablet_copy_client.cc:783] T 4bd8226227c348d5a5d4d8ff28959d27 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Starting download of 0 data blocks...
I20240503 01:59:58.105504 16590 tablet_copy_client.cc:647] T 4bd8226227c348d5a5d4d8ff28959d27 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Starting download of 1 WAL segments...
I20240503 01:59:58.112671 16590 tablet_copy_client.cc:523] T 4bd8226227c348d5a5d4d8ff28959d27 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 01:59:58.116709 16590 tablet_bootstrap.cc:492] T 4bd8226227c348d5a5d4d8ff28959d27 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:58.153491 16526 tablet_bootstrap.cc:492] T c4ac4ad7f9eb4723814f65ce96758c0e P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:58.154029 16526 tablet_bootstrap.cc:492] T c4ac4ad7f9eb4723814f65ce96758c0e P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap complete.
I20240503 01:59:58.156726 16526 ts_tablet_manager.cc:1367] T c4ac4ad7f9eb4723814f65ce96758c0e P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.053s user 0.030s sys 0.000s
I20240503 01:59:58.157400 16526 raft_consensus.cc:348] T c4ac4ad7f9eb4723814f65ce96758c0e P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:58.157665 16526 raft_consensus.cc:729] T c4ac4ad7f9eb4723814f65ce96758c0e P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:58.157846 16526 consensus_queue.cc:260] T c4ac4ad7f9eb4723814f65ce96758c0e P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:58.159559 16526 ts_tablet_manager.cc:1398] T c4ac4ad7f9eb4723814f65ce96758c0e P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 01:59:58.160095 6903 tablet_copy_service.cc:342] P 5dcc33cc621a4cbbae155f3230fb7c46: Request end of tablet copy session 12e8c683db054df3b75c6285b98a5ad8-c4ac4ad7f9eb4723814f65ce96758c0e received from {username='slave'} at 127.0.99.133:55147
I20240503 01:59:58.160189 6903 tablet_copy_service.cc:433] P 5dcc33cc621a4cbbae155f3230fb7c46: ending tablet copy session 12e8c683db054df3b75c6285b98a5ad8-c4ac4ad7f9eb4723814f65ce96758c0e on tablet c4ac4ad7f9eb4723814f65ce96758c0e with peer 12e8c683db054df3b75c6285b98a5ad8
I20240503 01:59:58.170142 16590 tablet_bootstrap.cc:492] T 4bd8226227c348d5a5d4d8ff28959d27 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 01:59:58.170604 16590 tablet_bootstrap.cc:492] T 4bd8226227c348d5a5d4d8ff28959d27 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap complete.
I20240503 01:59:58.170938 16590 ts_tablet_manager.cc:1367] T 4bd8226227c348d5a5d4d8ff28959d27 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.054s user 0.020s sys 0.009s
I20240503 01:59:58.171559 16590 raft_consensus.cc:348] T 4bd8226227c348d5a5d4d8ff28959d27 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:58.171906 16590 raft_consensus.cc:729] T 4bd8226227c348d5a5d4d8ff28959d27 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:58.172160 16590 consensus_queue.cc:260] T 4bd8226227c348d5a5d4d8ff28959d27 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:58.172814 16590 ts_tablet_manager.cc:1398] T 4bd8226227c348d5a5d4d8ff28959d27 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:58.173772 6903 tablet_copy_service.cc:342] P 5dcc33cc621a4cbbae155f3230fb7c46: Request end of tablet copy session b7062fbbc2b54fff9e99ec4ea5562c76-4bd8226227c348d5a5d4d8ff28959d27 received from {username='slave'} at 127.0.99.135:54599
I20240503 01:59:58.176776 6903 tablet_copy_service.cc:433] P 5dcc33cc621a4cbbae155f3230fb7c46: ending tablet copy session b7062fbbc2b54fff9e99ec4ea5562c76-4bd8226227c348d5a5d4d8ff28959d27 on tablet 4bd8226227c348d5a5d4d8ff28959d27 with peer b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 01:59:58.201356 16187 consensus_queue.cc:1035] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:58.212122 6740 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:58.220635 16609 consensus_queue.cc:1035] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:58.221575 6883 raft_consensus.cc:1261] T cef0e3380c4f47079780872d1515e77e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:58.222249 7275 raft_consensus.cc:1261] T cef0e3380c4f47079780872d1515e77e P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:58.224860 16187 consensus_queue.cc:1035] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:58.225258 16187 consensus_queue.cc:1035] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:58.226601 6861 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:58.230228 7125 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 01:59:58.235155 6739 catalog_manager.cc:5576] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943) does not have the latest schema for tablet cef0e3380c4f47079780872d1515e77e (table rebalancer_dynamic_table [id=63cb96ff9c214c618cca4cfade4c8614]). Expected version 1 got 0
I20240503 01:59:58.235857 9735 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:58.236114 9473 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:58.240540 6736 catalog_manager.cc:5576] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271) does not have the latest schema for tablet cef0e3380c4f47079780872d1515e77e (table rebalancer_dynamic_table [id=63cb96ff9c214c618cca4cfade4c8614]). Expected version 1 got 0
I20240503 01:59:58.244904 9342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 01:59:58.249328 6993 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 01:59:58.251971 16619 tablet.cc:1722] T cef0e3380c4f47079780872d1515e77e P 1d0e738c34af40fa9950c6f086619b3a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:58.251736 16618 tablet.cc:1722] T cef0e3380c4f47079780872d1515e77e P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:58.251482 16617 tablet.cc:1722] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:58.254547 6739 catalog_manager.cc:5576] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943) does not have the latest schema for tablet cef0e3380c4f47079780872d1515e77e (table rebalancer_dynamic_table [id=63cb96ff9c214c618cca4cfade4c8614]). Expected version 1 got 0
I20240503 01:59:58.255069 16618 tablet.cc:1620] T cef0e3380c4f47079780872d1515e77e P 5dcc33cc621a4cbbae155f3230fb7c46: MemRowSet was empty: no flush needed.
I20240503 01:59:58.252665 16619 tablet.cc:1620] T cef0e3380c4f47079780872d1515e77e P 1d0e738c34af40fa9950c6f086619b3a: MemRowSet was empty: no flush needed.
I20240503 01:59:58.255733 16617 tablet.cc:1620] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef: MemRowSet was empty: no flush needed.
I20240503 01:59:58.257784 6740 catalog_manager.cc:5576] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271) does not have the latest schema for tablet cef0e3380c4f47079780872d1515e77e (table rebalancer_dynamic_table [id=63cb96ff9c214c618cca4cfade4c8614]). Expected version 1 got 0
I20240503 01:59:58.267786 9605 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:58.267884 6739 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=63cb96ff9c214c618cca4cfade4c8614] alter complete (version 1)
I20240503 01:59:58.307264 7256 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:58.319890 6739 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:58.329706 6993 tablet_service.cc:1467] Processing CreateTablet for tablet fd29f3b059874b019153b64ef8aaaa48 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=7084fdc09131424f92874f08cc5ef36b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:58.330065 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd29f3b059874b019153b64ef8aaaa48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:58.329967 9735 tablet_service.cc:1467] Processing CreateTablet for tablet 2084fee295274ad89b407a933fb6070c (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=7084fdc09131424f92874f08cc5ef36b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:58.330125 6863 tablet_service.cc:1467] Processing CreateTablet for tablet 2084fee295274ad89b407a933fb6070c (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=7084fdc09131424f92874f08cc5ef36b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:58.330376 9735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2084fee295274ad89b407a933fb6070c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:58.330443 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2084fee295274ad89b407a933fb6070c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:58.331068 6862 tablet_service.cc:1467] Processing CreateTablet for tablet fd29f3b059874b019153b64ef8aaaa48 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=7084fdc09131424f92874f08cc5ef36b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:58.331305 6862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd29f3b059874b019153b64ef8aaaa48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:58.332544 16423 tablet_bootstrap.cc:492] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:58.333643 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 5bff4d54b9d54807b827bc776dce36c6 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=7084fdc09131424f92874f08cc5ef36b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:58.333719 16423 tablet_bootstrap.cc:654] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:58.333943 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bff4d54b9d54807b827bc776dce36c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:58.334020 16543 tablet_bootstrap.cc:492] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:58.334997 16543 tablet_bootstrap.cc:654] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:58.336153 16543 tablet_bootstrap.cc:492] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:58.336228 16543 ts_tablet_manager.cc:1367] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:58.336699 16543 raft_consensus.cc:348] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:58.336803 16543 raft_consensus.cc:374] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:58.336833 16543 raft_consensus.cc:729] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:58.336956 16543 consensus_queue.cc:260] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:58.337244 16543 ts_tablet_manager.cc:1398] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:58.337301 7254 tablet_service.cc:1467] Processing CreateTablet for tablet 2084fee295274ad89b407a933fb6070c (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=7084fdc09131424f92874f08cc5ef36b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:58.337625 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2084fee295274ad89b407a933fb6070c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:58.338346 9604 tablet_service.cc:1467] Processing CreateTablet for tablet fd29f3b059874b019153b64ef8aaaa48 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=7084fdc09131424f92874f08cc5ef36b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:58.338459 9605 tablet_service.cc:1467] Processing CreateTablet for tablet 5bff4d54b9d54807b827bc776dce36c6 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=7084fdc09131424f92874f08cc5ef36b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:58.338647 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd29f3b059874b019153b64ef8aaaa48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:58.338797 7256 tablet_service.cc:1467] Processing CreateTablet for tablet 5bff4d54b9d54807b827bc776dce36c6 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=7084fdc09131424f92874f08cc5ef36b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:58.339013 9605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bff4d54b9d54807b827bc776dce36c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:58.339054 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bff4d54b9d54807b827bc776dce36c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:58.340001 16544 tablet_bootstrap.cc:492] T 2084fee295274ad89b407a933fb6070c P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:58.341610 16544 tablet_bootstrap.cc:654] T 2084fee295274ad89b407a933fb6070c P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:58.341924 16543 tablet_bootstrap.cc:492] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:58.342998 16543 tablet_bootstrap.cc:654] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:58.344137 16543 tablet_bootstrap.cc:492] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:58.344231 16543 ts_tablet_manager.cc:1367] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:58.344689 16543 raft_consensus.cc:348] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:58.344789 16543 raft_consensus.cc:374] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:58.344818 16543 raft_consensus.cc:729] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:58.344933 16543 consensus_queue.cc:260] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:58.345232 16543 ts_tablet_manager.cc:1398] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:58.345409 16543 tablet_bootstrap.cc:492] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:58.346369 16543 tablet_bootstrap.cc:654] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:58.349435 16423 tablet_bootstrap.cc:492] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:58.349565 16423 ts_tablet_manager.cc:1367] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 01:59:58.350091 16423 raft_consensus.cc:348] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:58.350221 16423 raft_consensus.cc:374] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:58.350253 16423 raft_consensus.cc:729] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:58.350368 16423 consensus_queue.cc:260] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:58.350658 16423 ts_tablet_manager.cc:1398] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:58.353330 16622 tablet_bootstrap.cc:492] T fd29f3b059874b019153b64ef8aaaa48 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:58.353713 16544 tablet_bootstrap.cc:492] T 2084fee295274ad89b407a933fb6070c P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:58.353809 16544 ts_tablet_manager.cc:1367] T 2084fee295274ad89b407a933fb6070c P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240503 01:59:58.354342 16544 raft_consensus.cc:348] T 2084fee295274ad89b407a933fb6070c P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:58.354476 16544 raft_consensus.cc:374] T 2084fee295274ad89b407a933fb6070c P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:58.354565 16622 tablet_bootstrap.cc:654] T fd29f3b059874b019153b64ef8aaaa48 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:58.354566 16544 raft_consensus.cc:729] T 2084fee295274ad89b407a933fb6070c P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:58.354707 16544 consensus_queue.cc:260] T 2084fee295274ad89b407a933fb6070c P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:58.355044 16544 ts_tablet_manager.cc:1398] T 2084fee295274ad89b407a933fb6070c P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 01:59:58.355242 16544 tablet_bootstrap.cc:492] T 5bff4d54b9d54807b827bc776dce36c6 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:58.356216 16544 tablet_bootstrap.cc:654] T 5bff4d54b9d54807b827bc776dce36c6 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:58.358966 16543 tablet_bootstrap.cc:492] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:58.359074 16543 ts_tablet_manager.cc:1367] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20240503 01:59:58.359596 16543 raft_consensus.cc:348] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:58.359717 16543 raft_consensus.cc:374] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:58.359756 16543 raft_consensus.cc:729] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:58.359903 16543 consensus_queue.cc:260] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:58.360194 16543 ts_tablet_manager.cc:1398] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:58.364979 16626 tablet_bootstrap.cc:492] T 2084fee295274ad89b407a933fb6070c P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:58.366695 16626 tablet_bootstrap.cc:654] T 2084fee295274ad89b407a933fb6070c P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:58.370792 6740 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:58.371891 16622 tablet_bootstrap.cc:492] T fd29f3b059874b019153b64ef8aaaa48 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:58.372023 16622 ts_tablet_manager.cc:1367] T fd29f3b059874b019153b64ef8aaaa48 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240503 01:59:58.372504 16622 raft_consensus.cc:348] T fd29f3b059874b019153b64ef8aaaa48 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:58.372611 16622 raft_consensus.cc:374] T fd29f3b059874b019153b64ef8aaaa48 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:58.372839 16622 raft_consensus.cc:729] T fd29f3b059874b019153b64ef8aaaa48 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:58.373147 16622 consensus_queue.cc:260] T fd29f3b059874b019153b64ef8aaaa48 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:58.373729 16622 ts_tablet_manager.cc:1398] T fd29f3b059874b019153b64ef8aaaa48 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:58.373956 16622 tablet_bootstrap.cc:492] T 5bff4d54b9d54807b827bc776dce36c6 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:58.377038 16622 tablet_bootstrap.cc:654] T 5bff4d54b9d54807b827bc776dce36c6 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:58.379622 16622 tablet_bootstrap.cc:492] T 5bff4d54b9d54807b827bc776dce36c6 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:58.379737 16622 ts_tablet_manager.cc:1367] T 5bff4d54b9d54807b827bc776dce36c6 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 01:59:58.380306 16622 raft_consensus.cc:348] T 5bff4d54b9d54807b827bc776dce36c6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:58.380417 16622 raft_consensus.cc:374] T 5bff4d54b9d54807b827bc776dce36c6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:58.380446 16622 raft_consensus.cc:729] T 5bff4d54b9d54807b827bc776dce36c6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:58.380561 16622 consensus_queue.cc:260] T 5bff4d54b9d54807b827bc776dce36c6 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:58.380867 16622 ts_tablet_manager.cc:1398] T 5bff4d54b9d54807b827bc776dce36c6 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:58.384330 16544 tablet_bootstrap.cc:492] T 5bff4d54b9d54807b827bc776dce36c6 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:58.384438 16544 ts_tablet_manager.cc:1367] T 5bff4d54b9d54807b827bc776dce36c6 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20240503 01:59:58.385026 16544 raft_consensus.cc:348] T 5bff4d54b9d54807b827bc776dce36c6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:58.385138 16544 raft_consensus.cc:374] T 5bff4d54b9d54807b827bc776dce36c6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:58.385167 16544 raft_consensus.cc:729] T 5bff4d54b9d54807b827bc776dce36c6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:58.385284 16544 consensus_queue.cc:260] T 5bff4d54b9d54807b827bc776dce36c6 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:58.385550 16544 ts_tablet_manager.cc:1398] T 5bff4d54b9d54807b827bc776dce36c6 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:58.385814 16617 tablet.cc:1722] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:58.386173 16617 tablet.cc:1620] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef: MemRowSet was empty: no flush needed.
I20240503 01:59:58.386979 16626 tablet_bootstrap.cc:492] T 2084fee295274ad89b407a933fb6070c P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:58.387074 16626 ts_tablet_manager.cc:1367] T 2084fee295274ad89b407a933fb6070c P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240503 01:59:58.387516 16626 raft_consensus.cc:348] T 2084fee295274ad89b407a933fb6070c P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:58.387640 16626 raft_consensus.cc:374] T 2084fee295274ad89b407a933fb6070c P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:58.387681 16626 raft_consensus.cc:729] T 2084fee295274ad89b407a933fb6070c P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:58.387826 16626 consensus_queue.cc:260] T 2084fee295274ad89b407a933fb6070c P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:58.388120 16626 ts_tablet_manager.cc:1398] T 2084fee295274ad89b407a933fb6070c P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:58.389700 16618 tablet.cc:1722] T cef0e3380c4f47079780872d1515e77e P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:58.390065 16618 tablet.cc:1620] T cef0e3380c4f47079780872d1515e77e P 5dcc33cc621a4cbbae155f3230fb7c46: MemRowSet was empty: no flush needed.
I20240503 01:59:58.392762 16619 tablet.cc:1722] T cef0e3380c4f47079780872d1515e77e P 1d0e738c34af40fa9950c6f086619b3a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:58.393100 16619 tablet.cc:1620] T cef0e3380c4f47079780872d1515e77e P 1d0e738c34af40fa9950c6f086619b3a: MemRowSet was empty: no flush needed.
I20240503 01:59:58.398046 6736 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=63cb96ff9c214c618cca4cfade4c8614] alter complete (version 2)
I20240503 01:59:58.427706 9361 raft_consensus.cc:1203] T 2065dcd6cde84f9a928e4b817737dd42 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:58.474788 16401 raft_consensus.cc:482] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:58.474942 16401 raft_consensus.cc:504] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:58.475450 16401 leader_election.cc:288] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:58.475981 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bff4d54b9d54807b827bc776dce36c6" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:58.476154 7275 raft_consensus.cc:2454] T 5bff4d54b9d54807b827bc776dce36c6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:58.477018 6817 leader_election.cc:302] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:58.477545 9624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bff4d54b9d54807b827bc776dce36c6" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:58.477671 16401 raft_consensus.cc:2781] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:58.477703 9624 raft_consensus.cc:2454] T 5bff4d54b9d54807b827bc776dce36c6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:58.477746 16401 raft_consensus.cc:482] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:58.477792 16401 raft_consensus.cc:3037] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:58.479169 16401 raft_consensus.cc:504] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:58.479542 16401 leader_election.cc:288] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 01:59:58.480046 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bff4d54b9d54807b827bc776dce36c6" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:58.480183 7275 raft_consensus.cc:3037] T 5bff4d54b9d54807b827bc776dce36c6 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:58.481431 7275 raft_consensus.cc:2454] T 5bff4d54b9d54807b827bc776dce36c6 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:58.481858 6817 leader_election.cc:302] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:58.482064 9624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bff4d54b9d54807b827bc776dce36c6" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:58.482178 9624 raft_consensus.cc:3037] T 5bff4d54b9d54807b827bc776dce36c6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:58.483260 16401 raft_consensus.cc:2781] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:58.483357 16401 raft_consensus.cc:686] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 01:59:58.483317 9624 raft_consensus.cc:2454] T 5bff4d54b9d54807b827bc776dce36c6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:58.483703 16401 consensus_queue.cc:237] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:58.483800 16673 raft_consensus.cc:482] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:58.485244 16673 raft_consensus.cc:504] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:58.485941 16673 leader_election.cc:288] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:58.485925 9624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd29f3b059874b019153b64ef8aaaa48" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:58.486053 9624 raft_consensus.cc:2454] T fd29f3b059874b019153b64ef8aaaa48 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:58.486335 6818 leader_election.cc:302] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:58.486507 16401 raft_consensus.cc:2781] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:58.486601 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd29f3b059874b019153b64ef8aaaa48" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:58.486691 16401 raft_consensus.cc:482] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:58.486744 16401 raft_consensus.cc:3037] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:58.486754 7012 raft_consensus.cc:2454] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:58.487701 6739 catalog_manager.cc:5484] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:58.487874 16401 raft_consensus.cc:504] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:58.488335 16401 leader_election.cc:288] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 01:59:58.488919 9624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd29f3b059874b019153b64ef8aaaa48" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:58.489053 9624 raft_consensus.cc:3037] T fd29f3b059874b019153b64ef8aaaa48 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:58.489960 9624 raft_consensus.cc:2454] T fd29f3b059874b019153b64ef8aaaa48 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:58.490902 6818 leader_election.cc:302] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 01:59:58.491071 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd29f3b059874b019153b64ef8aaaa48" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:58.491350 7012 raft_consensus.cc:3037] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:58.491722 16673 raft_consensus.cc:482] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:58.491822 16673 raft_consensus.cc:504] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:58.492620 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2084fee295274ad89b407a933fb6070c" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:58.492779 7275 raft_consensus.cc:2454] T 2084fee295274ad89b407a933fb6070c P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:58.493183 6817 leader_election.cc:302] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 01:59:58.493253 7012 raft_consensus.cc:2454] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:58.493583 15668 raft_consensus.cc:2781] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:58.493614 16673 leader_election.cc:288] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:58.493710 16673 raft_consensus.cc:2781] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:58.493747 16673 raft_consensus.cc:482] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:58.493793 16673 raft_consensus.cc:3037] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:58.493752 15668 raft_consensus.cc:686] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 01:59:58.494172 15668 consensus_queue.cc:237] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 01:59:58.495038 16673 raft_consensus.cc:504] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:58.495150 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2084fee295274ad89b407a933fb6070c" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:58.495292 9757 raft_consensus.cc:2454] T 2084fee295274ad89b407a933fb6070c P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 01:59:58.495407 16673 leader_election.cc:288] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:58.495925 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2084fee295274ad89b407a933fb6070c" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:58.496057 7275 raft_consensus.cc:3037] T 2084fee295274ad89b407a933fb6070c P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:58.496338 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2084fee295274ad89b407a933fb6070c" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:58.496439 9757 raft_consensus.cc:3037] T 2084fee295274ad89b407a933fb6070c P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:58.497452 7275 raft_consensus.cc:2454] T 2084fee295274ad89b407a933fb6070c P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:58.497467 9757 raft_consensus.cc:2454] T 2084fee295274ad89b407a933fb6070c P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 01:59:58.497807 6817 leader_election.cc:302] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:58.498756 16673 raft_consensus.cc:2781] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:58.501493 16673 raft_consensus.cc:686] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 01:59:58.501668 16673 consensus_queue.cc:237] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 01:59:58.505296 6739 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:58.512765 16617 tablet.cc:1722] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:58.513080 16619 tablet.cc:1722] T cef0e3380c4f47079780872d1515e77e P 1d0e738c34af40fa9950c6f086619b3a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:58.515868 16618 tablet.cc:1722] T cef0e3380c4f47079780872d1515e77e P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:58.520089 6741 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=63cb96ff9c214c618cca4cfade4c8614] alter complete (version 3)
I20240503 01:59:58.523509 6740 catalog_manager.cc:5484] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 01:59:58.531579 6740 catalog_manager.cc:5484] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 01:59:58.596585 9342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 01:59:58.601987 7256 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:58.606951 9361 raft_consensus.cc:1203] T c4ac4ad7f9eb4723814f65ce96758c0e P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:58.610595 6993 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 01:59:58.611944 6863 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 01:59:58.623520 6739 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:58.623728 6739 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:58.626379 6739 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet cef0e3380c4f47079780872d1515e77e
I20240503 01:59:58.626963 9605 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:58.627244 9473 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:58.627229 7256 tablet_service.cc:1514] Processing DeleteTablet for tablet cef0e3380c4f47079780872d1515e77e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:58 UTC) from {username='slave'} at 127.0.0.1:45608
I20240503 01:59:58.627895 6993 tablet_service.cc:1514] Processing DeleteTablet for tablet cef0e3380c4f47079780872d1515e77e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:58 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 01:59:58.628017 6863 tablet_service.cc:1514] Processing DeleteTablet for tablet cef0e3380c4f47079780872d1515e77e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:58 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 01:59:58.628422 16689 tablet_replica.cc:330] T cef0e3380c4f47079780872d1515e77e P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 01:59:58.628794 16689 raft_consensus.cc:2229] T cef0e3380c4f47079780872d1515e77e P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:58.628974 16689 raft_consensus.cc:2258] T cef0e3380c4f47079780872d1515e77e P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:58.629961 16689 ts_tablet_manager.cc:1875] T cef0e3380c4f47079780872d1515e77e P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:58.636655 16689 ts_tablet_manager.cc:1888] T cef0e3380c4f47079780872d1515e77e P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:58.637130 16689 log.cc:1196] T cef0e3380c4f47079780872d1515e77e P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/cef0e3380c4f47079780872d1515e77e
I20240503 01:59:58.637663 16689 ts_tablet_manager.cc:1909] T cef0e3380c4f47079780872d1515e77e P 1d0e738c34af40fa9950c6f086619b3a: Deleting consensus metadata
I20240503 01:59:58.638504 16691 tablet_replica.cc:330] T cef0e3380c4f47079780872d1515e77e P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 01:59:58.638795 16691 raft_consensus.cc:2229] T cef0e3380c4f47079780872d1515e77e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:58.639012 16691 raft_consensus.cc:2258] T cef0e3380c4f47079780872d1515e77e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:58.639786 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet cef0e3380c4f47079780872d1515e77e (table rebalancer_dynamic_table_new_name [id=63cb96ff9c214c618cca4cfade4c8614]) successfully deleted
I20240503 01:59:58.640089 16691 ts_tablet_manager.cc:1875] T cef0e3380c4f47079780872d1515e77e P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:58.645027 7125 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 01:59:58.661659 16691 ts_tablet_manager.cc:1888] T cef0e3380c4f47079780872d1515e77e P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:58.661900 16691 log.cc:1196] T cef0e3380c4f47079780872d1515e77e P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/cef0e3380c4f47079780872d1515e77e
I20240503 01:59:58.662953 16691 ts_tablet_manager.cc:1909] T cef0e3380c4f47079780872d1515e77e P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 01:59:58.663803 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet cef0e3380c4f47079780872d1515e77e (table rebalancer_dynamic_table_new_name [id=63cb96ff9c214c618cca4cfade4c8614]) successfully deleted
I20240503 01:59:58.669760 9735 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:58.669735 16690 tablet_replica.cc:330] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 01:59:58.669996 16690 raft_consensus.cc:2229] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:58.670426 16690 raft_consensus.cc:2258] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:58.672605 16690 ts_tablet_manager.cc:1875] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:58.673342 9624 raft_consensus.cc:1203] T 4bd8226227c348d5a5d4d8ff28959d27 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 01:59:58.685487 16690 ts_tablet_manager.cc:1888] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:58.685587 16690 log.cc:1196] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/cef0e3380c4f47079780872d1515e77e
I20240503 01:59:58.685942 16690 ts_tablet_manager.cc:1909] T cef0e3380c4f47079780872d1515e77e P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 01:59:58.687252 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet cef0e3380c4f47079780872d1515e77e (table rebalancer_dynamic_table_new_name [id=63cb96ff9c214c618cca4cfade4c8614]) successfully deleted
I20240503 01:59:58.729367 6739 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:58.734500 6881 consensus_queue.cc:237] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:58.736203 9361 raft_consensus.cc:1261] T 2065dcd6cde84f9a928e4b817737dd42 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:58.736789 7014 raft_consensus.cc:1261] T 2065dcd6cde84f9a928e4b817737dd42 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:58.736773 16675 consensus_queue.cc:1035] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:58.737318 16675 consensus_queue.cc:1035] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:58.739356 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 652e8f55cfc245078cac08811d2cfd77 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9c752edb5c804520b5a2eb92b82e7963]), partition=
I20240503 01:59:58.739410 9473 tablet_service.cc:1467] Processing CreateTablet for tablet 652e8f55cfc245078cac08811d2cfd77 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9c752edb5c804520b5a2eb92b82e7963]), partition=
I20240503 01:59:58.739655 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 652e8f55cfc245078cac08811d2cfd77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:58.739727 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 652e8f55cfc245078cac08811d2cfd77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:58.739969 16401 raft_consensus.cc:2932] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:58.742452 16710 tablet_bootstrap.cc:492] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:58.742760 7125 tablet_service.cc:1467] Processing CreateTablet for tablet 652e8f55cfc245078cac08811d2cfd77 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9c752edb5c804520b5a2eb92b82e7963]), partition=
I20240503 01:59:58.743014 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 652e8f55cfc245078cac08811d2cfd77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:58.743618 7014 raft_consensus.cc:2932] T 2065dcd6cde84f9a928e4b817737dd42 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:58.744091 9363 raft_consensus.cc:2932] T 2065dcd6cde84f9a928e4b817737dd42 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:58.744171 6741 catalog_manager.cc:5484] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:58.749065 16710 tablet_bootstrap.cc:654] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:58.750634 7124 tablet_service.cc:1514] Processing DeleteTablet for tablet 2065dcd6cde84f9a928e4b817737dd42 with delete_type TABLET_DATA_TOMBSTONED (TS c688f3df56ac4b6d9b0c30fa4cc8ca2a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:58.751068 16713 tablet_replica.cc:330] T 2065dcd6cde84f9a928e4b817737dd42 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:58.751475 16714 tablet_bootstrap.cc:492] T 652e8f55cfc245078cac08811d2cfd77 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:58.751490 16713 raft_consensus.cc:2229] T 2065dcd6cde84f9a928e4b817737dd42 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:58.751847 16713 raft_consensus.cc:2258] T 2065dcd6cde84f9a928e4b817737dd42 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:58.752578 16713 ts_tablet_manager.cc:1875] T 2065dcd6cde84f9a928e4b817737dd42 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:58.753261 16714 tablet_bootstrap.cc:654] T 652e8f55cfc245078cac08811d2cfd77 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:58.756086 16713 ts_tablet_manager.cc:1888] T 2065dcd6cde84f9a928e4b817737dd42 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 01:59:58.756179 16713 log.cc:1196] T 2065dcd6cde84f9a928e4b817737dd42 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/2065dcd6cde84f9a928e4b817737dd42
I20240503 01:59:58.756843 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet 2065dcd6cde84f9a928e4b817737dd42 (table rebalance_test_table_7 [id=3428bac98e0c49908450b3b51cb9d18c]) successfully deleted
I20240503 01:59:58.757294 16711 tablet_bootstrap.cc:492] T 652e8f55cfc245078cac08811d2cfd77 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:58.758292 16711 tablet_bootstrap.cc:654] T 652e8f55cfc245078cac08811d2cfd77 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:58.759877 16711 tablet_bootstrap.cc:492] T 652e8f55cfc245078cac08811d2cfd77 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:58.760053 16711 ts_tablet_manager.cc:1367] T 652e8f55cfc245078cac08811d2cfd77 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:58.760583 16711 raft_consensus.cc:348] T 652e8f55cfc245078cac08811d2cfd77 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:58.760847 16711 raft_consensus.cc:374] T 652e8f55cfc245078cac08811d2cfd77 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:58.760932 16711 raft_consensus.cc:729] T 652e8f55cfc245078cac08811d2cfd77 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:58.761147 16711 consensus_queue.cc:260] T 652e8f55cfc245078cac08811d2cfd77 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:58.762063 16711 ts_tablet_manager.cc:1398] T 652e8f55cfc245078cac08811d2cfd77 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:58.764498 16714 tablet_bootstrap.cc:492] T 652e8f55cfc245078cac08811d2cfd77 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:58.764698 16714 ts_tablet_manager.cc:1367] T 652e8f55cfc245078cac08811d2cfd77 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240503 01:59:58.765553 16710 tablet_bootstrap.cc:492] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:58.765676 16710 ts_tablet_manager.cc:1367] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240503 01:59:58.766506 16710 raft_consensus.cc:348] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:58.766741 16710 raft_consensus.cc:374] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:58.766804 16710 raft_consensus.cc:729] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:58.766963 16710 consensus_queue.cc:260] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:58.767403 16710 ts_tablet_manager.cc:1398] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 01:59:58.767001 16714 raft_consensus.cc:348] T 652e8f55cfc245078cac08811d2cfd77 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:58.768512 16714 raft_consensus.cc:374] T 652e8f55cfc245078cac08811d2cfd77 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:58.768618 16714 raft_consensus.cc:729] T 652e8f55cfc245078cac08811d2cfd77 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:58.768857 16714 consensus_queue.cc:260] T 652e8f55cfc245078cac08811d2cfd77 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:58.769634 16714 ts_tablet_manager.cc:1398] T 652e8f55cfc245078cac08811d2cfd77 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 01:59:58.866355 16401 consensus_queue.cc:1035] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:58.872160 16675 consensus_queue.cc:1035] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:58.913472 16467 raft_consensus.cc:482] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:58.913679 16467 raft_consensus.cc:504] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:58.914440 16467 leader_election.cc:288] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:58.915053 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "652e8f55cfc245078cac08811d2cfd77" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:58.915227 9363 raft_consensus.cc:2454] T 652e8f55cfc245078cac08811d2cfd77 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:58.915283 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "652e8f55cfc245078cac08811d2cfd77" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:58.915403 7145 raft_consensus.cc:2454] T 652e8f55cfc245078cac08811d2cfd77 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:58.916551 9429 leader_election.cc:302] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:58.916775 16467 raft_consensus.cc:2781] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:58.916846 16467 raft_consensus.cc:482] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:58.916884 16467 raft_consensus.cc:3037] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:58.918201 16467 raft_consensus.cc:504] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:58.918638 16467 leader_election.cc:288] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Requested vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:58.919095 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "652e8f55cfc245078cac08811d2cfd77" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:58.919226 7145 raft_consensus.cc:3037] T 652e8f55cfc245078cac08811d2cfd77 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:58.919466 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "652e8f55cfc245078cac08811d2cfd77" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:58.919622 9363 raft_consensus.cc:3037] T 652e8f55cfc245078cac08811d2cfd77 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:58.920504 7145 raft_consensus.cc:2454] T 652e8f55cfc245078cac08811d2cfd77 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:58.920689 9363 raft_consensus.cc:2454] T 652e8f55cfc245078cac08811d2cfd77 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:58.920898 9429 leader_election.cc:302] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:58.923240 16467 raft_consensus.cc:2781] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:58.923417 16467 raft_consensus.cc:686] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Becoming Leader. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Running, Role: LEADER
I20240503 01:59:58.923892 16467 consensus_queue.cc:237] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:58.928206 6734 catalog_manager.cc:5484] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: term changed from 0 to 1, leader changed from <none> to bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:58.960655 16675 consensus_queue.cc:1035] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:58.960976 6863 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 01:59:58.961045 9473 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:58.974468 9342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 01:59:58.979488 9735 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:58.980145 16401 consensus_queue.cc:1035] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:58.995200 7256 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:58.996722 6993 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 01:59:59.000694 9605 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:59.011914 7125 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 01:59:59.019847 16673 consensus_queue.cc:1035] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:59.049885 6734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:59.056529 16675 consensus_queue.cc:1035] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:59.062636 9735 tablet_service.cc:1467] Processing CreateTablet for tablet 5576de720be6425a80b49499506c363f (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=8ade9b2b5b1a4a56a57cd8db33d169d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:59.062711 7125 tablet_service.cc:1467] Processing CreateTablet for tablet 5576de720be6425a80b49499506c363f (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=8ade9b2b5b1a4a56a57cd8db33d169d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:59.062968 9735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5576de720be6425a80b49499506c363f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.063042 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5576de720be6425a80b49499506c363f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.063432 9736 tablet_service.cc:1467] Processing CreateTablet for tablet 38f126abb17c4654ae3cabe688d513d5 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=8ade9b2b5b1a4a56a57cd8db33d169d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:59.063665 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38f126abb17c4654ae3cabe688d513d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.065213 16786 tablet_bootstrap.cc:492] T 5576de720be6425a80b49499506c363f P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:59.065646 7256 tablet_service.cc:1467] Processing CreateTablet for tablet 168c7491d50c44b39c5698c3f9d2e98f (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=8ade9b2b5b1a4a56a57cd8db33d169d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:59.065690 6993 tablet_service.cc:1467] Processing CreateTablet for tablet 5576de720be6425a80b49499506c363f (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=8ade9b2b5b1a4a56a57cd8db33d169d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:59.065950 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 168c7491d50c44b39c5698c3f9d2e98f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.065965 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5576de720be6425a80b49499506c363f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.066675 16714 tablet_bootstrap.cc:492] T 5576de720be6425a80b49499506c363f P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:59.067724 16714 tablet_bootstrap.cc:654] T 5576de720be6425a80b49499506c363f P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.068603 16787 tablet_bootstrap.cc:492] T 5576de720be6425a80b49499506c363f P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:59.068920 6992 tablet_service.cc:1467] Processing CreateTablet for tablet 38f126abb17c4654ae3cabe688d513d5 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=8ade9b2b5b1a4a56a57cd8db33d169d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:59.069204 6992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38f126abb17c4654ae3cabe688d513d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.069319 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:59.069351 6994 tablet_service.cc:1467] Processing CreateTablet for tablet 168c7491d50c44b39c5698c3f9d2e98f (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=8ade9b2b5b1a4a56a57cd8db33d169d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:59.069571 6994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 168c7491d50c44b39c5698c3f9d2e98f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.069918 16787 tablet_bootstrap.cc:654] T 5576de720be6425a80b49499506c363f P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.071746 6863 tablet_service.cc:1467] Processing CreateTablet for tablet 168c7491d50c44b39c5698c3f9d2e98f (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=8ade9b2b5b1a4a56a57cd8db33d169d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:59.072029 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 168c7491d50c44b39c5698c3f9d2e98f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.073685 16790 tablet_bootstrap.cc:492] T 168c7491d50c44b39c5698c3f9d2e98f P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:59.074860 16790 tablet_bootstrap.cc:654] T 168c7491d50c44b39c5698c3f9d2e98f P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.076164 16714 tablet_bootstrap.cc:492] T 5576de720be6425a80b49499506c363f P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:59.076294 16714 ts_tablet_manager.cc:1367] T 5576de720be6425a80b49499506c363f P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240503 01:59:59.076509 16790 tablet_bootstrap.cc:492] T 168c7491d50c44b39c5698c3f9d2e98f P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:59.076587 16790 ts_tablet_manager.cc:1367] T 168c7491d50c44b39c5698c3f9d2e98f P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:59.076860 16714 raft_consensus.cc:348] T 5576de720be6425a80b49499506c363f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:59.076977 16714 raft_consensus.cc:374] T 5576de720be6425a80b49499506c363f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.077013 16714 raft_consensus.cc:729] T 5576de720be6425a80b49499506c363f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.077025 16790 raft_consensus.cc:348] T 168c7491d50c44b39c5698c3f9d2e98f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:59.077124 16790 raft_consensus.cc:374] T 168c7491d50c44b39c5698c3f9d2e98f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.077150 16790 raft_consensus.cc:729] T 168c7491d50c44b39c5698c3f9d2e98f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.077140 16714 consensus_queue.cc:260] T 5576de720be6425a80b49499506c363f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:59.077262 16790 consensus_queue.cc:260] T 168c7491d50c44b39c5698c3f9d2e98f P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:59.077477 16714 ts_tablet_manager.cc:1398] T 5576de720be6425a80b49499506c363f P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:59.077512 16790 ts_tablet_manager.cc:1398] T 168c7491d50c44b39c5698c3f9d2e98f P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:59.082314 7145 raft_consensus.cc:1261] T 652e8f55cfc245078cac08811d2cfd77 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:59.082827 9363 raft_consensus.cc:1261] T 652e8f55cfc245078cac08811d2cfd77 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:59.082893 16467 consensus_queue.cc:1035] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:59.084726 16467 consensus_queue.cc:1035] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:59.087543 16786 tablet_bootstrap.cc:654] T 5576de720be6425a80b49499506c363f P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.089627 6734 catalog_manager.cc:5576] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387) does not have the latest schema for tablet 652e8f55cfc245078cac08811d2cfd77 (table rebalancer_dynamic_table [id=9c752edb5c804520b5a2eb92b82e7963]). Expected version 1 got 0
I20240503 01:59:59.093751 6738 catalog_manager.cc:5576] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387) does not have the latest schema for tablet 652e8f55cfc245078cac08811d2cfd77 (table rebalancer_dynamic_table [id=9c752edb5c804520b5a2eb92b82e7963]). Expected version 1 got 0
I20240503 01:59:59.095124 7125 tablet_service.cc:1467] Processing CreateTablet for tablet 38f126abb17c4654ae3cabe688d513d5 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=8ade9b2b5b1a4a56a57cd8db33d169d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:59.095454 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38f126abb17c4654ae3cabe688d513d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.096515 6734 catalog_manager.cc:5576] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979) does not have the latest schema for tablet 652e8f55cfc245078cac08811d2cfd77 (table rebalancer_dynamic_table [id=9c752edb5c804520b5a2eb92b82e7963]). Expected version 1 got 0
I20240503 01:59:59.097867 16786 tablet_bootstrap.cc:492] T 5576de720be6425a80b49499506c363f P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:59.097970 16786 ts_tablet_manager.cc:1367] T 5576de720be6425a80b49499506c363f P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.000s
I20240503 01:59:59.098469 16786 raft_consensus.cc:348] T 5576de720be6425a80b49499506c363f P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:59.098659 16786 raft_consensus.cc:374] T 5576de720be6425a80b49499506c363f P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.098717 16786 raft_consensus.cc:729] T 5576de720be6425a80b49499506c363f P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.098891 16714 tablet_bootstrap.cc:492] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:59.098872 16786 consensus_queue.cc:260] T 5576de720be6425a80b49499506c363f P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:59.099982 16714 tablet_bootstrap.cc:654] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.100028 16786 ts_tablet_manager.cc:1398] T 5576de720be6425a80b49499506c363f P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 01:59:59.100206 16786 tablet_bootstrap.cc:492] T 38f126abb17c4654ae3cabe688d513d5 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 01:59:59.101487 16786 tablet_bootstrap.cc:654] T 38f126abb17c4654ae3cabe688d513d5 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.105969 16798 tablet.cc:1722] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:59.106839 16800 tablet.cc:1722] T 652e8f55cfc245078cac08811d2cfd77 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:59.106997 16787 tablet_bootstrap.cc:492] T 5576de720be6425a80b49499506c363f P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:59.107121 16787 ts_tablet_manager.cc:1367] T 5576de720be6425a80b49499506c363f P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.000s
I20240503 01:59:59.107179 16800 tablet.cc:1620] T 652e8f55cfc245078cac08811d2cfd77 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: MemRowSet was empty: no flush needed.
I20240503 01:59:59.107647 16787 raft_consensus.cc:348] T 5576de720be6425a80b49499506c363f P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:59.107767 16787 raft_consensus.cc:374] T 5576de720be6425a80b49499506c363f P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.107797 16787 raft_consensus.cc:729] T 5576de720be6425a80b49499506c363f P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.107914 16798 tablet.cc:1620] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:59.107923 16787 consensus_queue.cc:260] T 5576de720be6425a80b49499506c363f P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:59.108215 16787 ts_tablet_manager.cc:1398] T 5576de720be6425a80b49499506c363f P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:59.108361 16787 tablet_bootstrap.cc:492] T 168c7491d50c44b39c5698c3f9d2e98f P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:59.109373 16787 tablet_bootstrap.cc:654] T 168c7491d50c44b39c5698c3f9d2e98f P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.112083 6738 catalog_manager.cc:5576] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979) does not have the latest schema for tablet 652e8f55cfc245078cac08811d2cfd77 (table rebalancer_dynamic_table [id=9c752edb5c804520b5a2eb92b82e7963]). Expected version 1 got 0
I20240503 01:59:59.114306 16787 tablet_bootstrap.cc:492] T 168c7491d50c44b39c5698c3f9d2e98f P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:59.114414 16787 ts_tablet_manager.cc:1367] T 168c7491d50c44b39c5698c3f9d2e98f P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 01:59:59.114962 16787 raft_consensus.cc:348] T 168c7491d50c44b39c5698c3f9d2e98f P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:59.100797 16799 tablet_bootstrap.cc:492] T 168c7491d50c44b39c5698c3f9d2e98f P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:59.115073 16787 raft_consensus.cc:374] T 168c7491d50c44b39c5698c3f9d2e98f P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.115106 16787 raft_consensus.cc:729] T 168c7491d50c44b39c5698c3f9d2e98f P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.115236 16787 consensus_queue.cc:260] T 168c7491d50c44b39c5698c3f9d2e98f P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:59.115567 16787 ts_tablet_manager.cc:1398] T 168c7491d50c44b39c5698c3f9d2e98f P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:59.116884 16799 tablet_bootstrap.cc:654] T 168c7491d50c44b39c5698c3f9d2e98f P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.118317 16714 tablet_bootstrap.cc:492] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:59.118508 16714 ts_tablet_manager.cc:1367] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240503 01:59:59.118551 16803 tablet.cc:1722] T 652e8f55cfc245078cac08811d2cfd77 P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:59.118893 16803 tablet.cc:1620] T 652e8f55cfc245078cac08811d2cfd77 P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 01:59:59.118939 16787 tablet_bootstrap.cc:492] T 38f126abb17c4654ae3cabe688d513d5 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 01:59:59.119102 16714 raft_consensus.cc:348] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:59.119218 16714 raft_consensus.cc:374] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.119261 16714 raft_consensus.cc:729] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.119382 16714 consensus_queue.cc:260] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:59.119676 16714 ts_tablet_manager.cc:1398] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:59.121212 16787 tablet_bootstrap.cc:654] T 38f126abb17c4654ae3cabe688d513d5 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.124194 16786 tablet_bootstrap.cc:492] T 38f126abb17c4654ae3cabe688d513d5 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 01:59:59.124315 16786 ts_tablet_manager.cc:1367] T 38f126abb17c4654ae3cabe688d513d5 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.001s
I20240503 01:59:59.125005 16786 raft_consensus.cc:348] T 38f126abb17c4654ae3cabe688d513d5 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:59.125185 16786 raft_consensus.cc:374] T 38f126abb17c4654ae3cabe688d513d5 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.125248 16786 raft_consensus.cc:729] T 38f126abb17c4654ae3cabe688d513d5 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.125416 16786 consensus_queue.cc:260] T 38f126abb17c4654ae3cabe688d513d5 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:59.126660 16786 ts_tablet_manager.cc:1398] T 38f126abb17c4654ae3cabe688d513d5 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 01:59:59.134588 16787 tablet_bootstrap.cc:492] T 38f126abb17c4654ae3cabe688d513d5 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 01:59:59.134702 16787 ts_tablet_manager.cc:1367] T 38f126abb17c4654ae3cabe688d513d5 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.003s
I20240503 01:59:59.135167 16787 raft_consensus.cc:348] T 38f126abb17c4654ae3cabe688d513d5 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:59.135270 16787 raft_consensus.cc:374] T 38f126abb17c4654ae3cabe688d513d5 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.135299 16787 raft_consensus.cc:729] T 38f126abb17c4654ae3cabe688d513d5 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.135406 16787 consensus_queue.cc:260] T 38f126abb17c4654ae3cabe688d513d5 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:59.135654 16787 ts_tablet_manager.cc:1398] T 38f126abb17c4654ae3cabe688d513d5 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:59.139142 6737 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=9c752edb5c804520b5a2eb92b82e7963] alter complete (version 1)
I20240503 01:59:59.148823 16799 tablet_bootstrap.cc:492] T 168c7491d50c44b39c5698c3f9d2e98f P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:59.154654 16799 ts_tablet_manager.cc:1367] T 168c7491d50c44b39c5698c3f9d2e98f P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.000s
I20240503 01:59:59.155256 16799 raft_consensus.cc:348] T 168c7491d50c44b39c5698c3f9d2e98f P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:59.155395 16799 raft_consensus.cc:374] T 168c7491d50c44b39c5698c3f9d2e98f P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.155445 16799 raft_consensus.cc:729] T 168c7491d50c44b39c5698c3f9d2e98f P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.155597 16799 consensus_queue.cc:260] T 168c7491d50c44b39c5698c3f9d2e98f P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:59.155925 16799 ts_tablet_manager.cc:1398] T 168c7491d50c44b39c5698c3f9d2e98f P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:59.156164 16636 raft_consensus.cc:482] T 168c7491d50c44b39c5698c3f9d2e98f P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:59.156277 16636 raft_consensus.cc:504] T 168c7491d50c44b39c5698c3f9d2e98f P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:59.156729 16636 leader_election.cc:288] T 168c7491d50c44b39c5698c3f9d2e98f P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:59.157490 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "168c7491d50c44b39c5698c3f9d2e98f" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:59.157627 6881 raft_consensus.cc:2454] T 168c7491d50c44b39c5698c3f9d2e98f P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:59.157927 6950 leader_election.cc:302] T 168c7491d50c44b39c5698c3f9d2e98f P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:59.158115 16636 raft_consensus.cc:2781] T 168c7491d50c44b39c5698c3f9d2e98f P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:59.158185 16636 raft_consensus.cc:482] T 168c7491d50c44b39c5698c3f9d2e98f P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:59.158221 16636 raft_consensus.cc:3037] T 168c7491d50c44b39c5698c3f9d2e98f P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.158733 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "168c7491d50c44b39c5698c3f9d2e98f" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 01:59:59.158881 7274 raft_consensus.cc:2454] T 168c7491d50c44b39c5698c3f9d2e98f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:59.159618 16636 raft_consensus.cc:504] T 168c7491d50c44b39c5698c3f9d2e98f P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:59.160022 16636 leader_election.cc:288] T 168c7491d50c44b39c5698c3f9d2e98f P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:59.160466 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "168c7491d50c44b39c5698c3f9d2e98f" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 01:59:59.160585 7274 raft_consensus.cc:3037] T 168c7491d50c44b39c5698c3f9d2e98f P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.161700 7274 raft_consensus.cc:2454] T 168c7491d50c44b39c5698c3f9d2e98f P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:59.162312 6948 leader_election.cc:302] T 168c7491d50c44b39c5698c3f9d2e98f P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 01:59:59.162755 16636 raft_consensus.cc:2781] T 168c7491d50c44b39c5698c3f9d2e98f P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:59.162838 16636 raft_consensus.cc:686] T 168c7491d50c44b39c5698c3f9d2e98f P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:59.162963 16636 consensus_queue.cc:237] T 168c7491d50c44b39c5698c3f9d2e98f P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:59.165062 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "168c7491d50c44b39c5698c3f9d2e98f" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:59.165220 6881 raft_consensus.cc:3037] T 168c7491d50c44b39c5698c3f9d2e98f P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.166262 6881 raft_consensus.cc:2454] T 168c7491d50c44b39c5698c3f9d2e98f P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:59.168709 6737 catalog_manager.cc:5484] T 168c7491d50c44b39c5698c3f9d2e98f P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:59.187279 9494 consensus_queue.cc:237] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:59.189352 6881 raft_consensus.cc:1261] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:59.189352 9363 raft_consensus.cc:1261] T 3e824fcc9aac449d9373655b304010e2 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 01:59:59.189985 16819 consensus_queue.cc:1035] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:59.190264 16467 consensus_queue.cc:1035] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 01:59:59.193795 15096 raft_consensus.cc:2932] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:59.195319 9361 raft_consensus.cc:2932] T 3e824fcc9aac449d9373655b304010e2 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:59.195319 6881 raft_consensus.cc:2932] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:59.197659 6734 catalog_manager.cc:5484] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: config changed from index 2 to 3, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:59.202586 7125 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e824fcc9aac449d9373655b304010e2 with delete_type TABLET_DATA_TOMBSTONED (TS c688f3df56ac4b6d9b0c30fa4cc8ca2a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:59.202906 16713 tablet_replica.cc:330] T 3e824fcc9aac449d9373655b304010e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:59.203068 16713 raft_consensus.cc:2229] T 3e824fcc9aac449d9373655b304010e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:59.203202 16713 raft_consensus.cc:2258] T 3e824fcc9aac449d9373655b304010e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:59.203637 16713 ts_tablet_manager.cc:1875] T 3e824fcc9aac449d9373655b304010e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:59.206110 16713 ts_tablet_manager.cc:1888] T 3e824fcc9aac449d9373655b304010e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 01:59:59.206187 16713 log.cc:1196] T 3e824fcc9aac449d9373655b304010e2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/3e824fcc9aac449d9373655b304010e2
I20240503 01:59:59.209393 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet 3e824fcc9aac449d9373655b304010e2 (table rebalancer_extra_table_10 [id=15afeaf3f27142ebaf9ef93cac9e4dd6]) successfully deleted
I20240503 01:59:59.228284 16636 raft_consensus.cc:482] T 5576de720be6425a80b49499506c363f P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:59.228642 16636 raft_consensus.cc:504] T 5576de720be6425a80b49499506c363f P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:59.229188 16636 leader_election.cc:288] T 5576de720be6425a80b49499506c363f P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:59.230173 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5576de720be6425a80b49499506c363f" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:59.230396 9755 raft_consensus.cc:2454] T 5576de720be6425a80b49499506c363f P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:59.230434 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5576de720be6425a80b49499506c363f" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 01:59:59.230633 7145 raft_consensus.cc:2454] T 5576de720be6425a80b49499506c363f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 01:59:59.230948 6949 leader_election.cc:302] T 5576de720be6425a80b49499506c363f P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:59.231149 16636 raft_consensus.cc:2781] T 5576de720be6425a80b49499506c363f P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:59.231204 16636 raft_consensus.cc:482] T 5576de720be6425a80b49499506c363f P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:59.231241 16636 raft_consensus.cc:3037] T 5576de720be6425a80b49499506c363f P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.231570 16780 raft_consensus.cc:482] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:59.231714 16780 raft_consensus.cc:504] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:59.232226 16780 leader_election.cc:288] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:59.232616 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38f126abb17c4654ae3cabe688d513d5" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 01:59:59.232661 16636 raft_consensus.cc:504] T 5576de720be6425a80b49499506c363f P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:59.232750 9755 raft_consensus.cc:2454] T 38f126abb17c4654ae3cabe688d513d5 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:59.233024 16636 leader_election.cc:288] T 5576de720be6425a80b49499506c363f P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 01:59:59.233150 7079 leader_election.cc:302] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:59.233481 15159 raft_consensus.cc:2781] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:59.233541 15159 raft_consensus.cc:482] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:59.233618 15159 raft_consensus.cc:3037] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.234115 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38f126abb17c4654ae3cabe688d513d5" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 01:59:59.234258 7012 raft_consensus.cc:2454] T 38f126abb17c4654ae3cabe688d513d5 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 01:59:59.234884 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5576de720be6425a80b49499506c363f" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 01:59:59.235044 7145 raft_consensus.cc:3037] T 5576de720be6425a80b49499506c363f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.235087 15159 raft_consensus.cc:504] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:59.235512 15159 leader_election.cc:288] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Requested vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 01:59:59.236171 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38f126abb17c4654ae3cabe688d513d5" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:59.236285 7145 raft_consensus.cc:2454] T 5576de720be6425a80b49499506c363f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:59.236295 9757 raft_consensus.cc:3037] T 38f126abb17c4654ae3cabe688d513d5 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.236788 6949 leader_election.cc:302] T 5576de720be6425a80b49499506c363f P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 01:59:59.237179 14833 raft_consensus.cc:2781] T 5576de720be6425a80b49499506c363f P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:59.237167 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38f126abb17c4654ae3cabe688d513d5" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 01:59:59.237246 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5576de720be6425a80b49499506c363f" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 01:59:59.237344 7012 raft_consensus.cc:3037] T 38f126abb17c4654ae3cabe688d513d5 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.237372 9755 raft_consensus.cc:3037] T 5576de720be6425a80b49499506c363f P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.237404 14833 raft_consensus.cc:686] T 5576de720be6425a80b49499506c363f P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 01:59:59.237452 9757 raft_consensus.cc:2454] T 38f126abb17c4654ae3cabe688d513d5 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:59.237594 14833 consensus_queue.cc:237] T 5576de720be6425a80b49499506c363f P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:59.238386 9755 raft_consensus.cc:2454] T 5576de720be6425a80b49499506c363f P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 01:59:59.238982 7012 raft_consensus.cc:2454] T 38f126abb17c4654ae3cabe688d513d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 01:59:59.239557 7079 leader_election.cc:302] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 01:59:59.239828 16780 raft_consensus.cc:2781] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:59.239944 16780 raft_consensus.cc:686] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Becoming Leader. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Running, Role: LEADER
I20240503 01:59:59.240114 16780 consensus_queue.cc:237] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 01:59:59.241843 6737 catalog_manager.cc:5484] T 5576de720be6425a80b49499506c363f P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:59.243764 6732 catalog_manager.cc:5484] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 01:59:59.245906 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 01:59:59.254357 16798 tablet.cc:1722] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:59.254783 16798 tablet.cc:1620] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:59.255138 16803 tablet.cc:1722] T 652e8f55cfc245078cac08811d2cfd77 P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:59.255434 16803 tablet.cc:1620] T 652e8f55cfc245078cac08811d2cfd77 P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 01:59:59.255944 16800 tablet.cc:1722] T 652e8f55cfc245078cac08811d2cfd77 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 01:59:59.256279 16800 tablet.cc:1620] T 652e8f55cfc245078cac08811d2cfd77 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: MemRowSet was empty: no flush needed.
I20240503 01:59:59.263448 6732 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=9c752edb5c804520b5a2eb92b82e7963] alter complete (version 2)
I20240503 01:59:59.367230 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 01:59:59.375488 16798 tablet.cc:1722] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:59.376021 16803 tablet.cc:1722] T 652e8f55cfc245078cac08811d2cfd77 P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:59.383296 6734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=9c752edb5c804520b5a2eb92b82e7963] alter complete (version 3)
I20240503 01:59:59.392251 7256 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:59.392683 9605 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:59.395434 16800 tablet.cc:1722] T 652e8f55cfc245078cac08811d2cfd77 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 01:59:59.403579 6992 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 01:59:59.412758 9736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:59.414840 9342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 01:59:59.416013 6863 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 01:59:59.418948 7125 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 01:59:59.426892 9473 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:59.487541 6739 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:59.487798 6739 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 01:59:59.494201 6739 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 652e8f55cfc245078cac08811d2cfd77
I20240503 01:59:59.495591 9473 tablet_service.cc:1514] Processing DeleteTablet for tablet 652e8f55cfc245078cac08811d2cfd77 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:59 UTC) from {username='slave'} at 127.0.0.1:54678
I20240503 01:59:59.495795 9342 tablet_service.cc:1514] Processing DeleteTablet for tablet 652e8f55cfc245078cac08811d2cfd77 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:59 UTC) from {username='slave'} at 127.0.0.1:36040
I20240503 01:59:59.496253 7125 tablet_service.cc:1514] Processing DeleteTablet for tablet 652e8f55cfc245078cac08811d2cfd77 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 01:59:59 UTC) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:59.496575 16713 tablet_replica.cc:330] T 652e8f55cfc245078cac08811d2cfd77 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:59.496747 16713 raft_consensus.cc:2229] T 652e8f55cfc245078cac08811d2cfd77 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:59.496898 16713 raft_consensus.cc:2258] T 652e8f55cfc245078cac08811d2cfd77 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:59.498301 16713 ts_tablet_manager.cc:1875] T 652e8f55cfc245078cac08811d2cfd77 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:59.502954 16713 ts_tablet_manager.cc:1888] T 652e8f55cfc245078cac08811d2cfd77 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:59.503095 16713 log.cc:1196] T 652e8f55cfc245078cac08811d2cfd77 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/652e8f55cfc245078cac08811d2cfd77
I20240503 01:59:59.503401 16713 ts_tablet_manager.cc:1909] T 652e8f55cfc245078cac08811d2cfd77 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting consensus metadata
I20240503 01:59:59.504400 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet 652e8f55cfc245078cac08811d2cfd77 (table rebalancer_dynamic_table_new_name [id=9c752edb5c804520b5a2eb92b82e7963]) successfully deleted
I20240503 01:59:59.504539 16901 tablet_replica.cc:330] T 652e8f55cfc245078cac08811d2cfd77 P 12e8c683db054df3b75c6285b98a5ad8: stopping tablet replica
I20240503 01:59:59.504724 16901 raft_consensus.cc:2229] T 652e8f55cfc245078cac08811d2cfd77 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:59.504840 16901 raft_consensus.cc:2258] T 652e8f55cfc245078cac08811d2cfd77 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:59.505690 16901 ts_tablet_manager.cc:1875] T 652e8f55cfc245078cac08811d2cfd77 P 12e8c683db054df3b75c6285b98a5ad8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:59.506827 16900 tablet_replica.cc:330] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a: stopping tablet replica
I20240503 01:59:59.506999 16900 raft_consensus.cc:2229] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Raft consensus shutting down.
I20240503 01:59:59.507233 16900 raft_consensus.cc:2258] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:59.508683 16900 ts_tablet_manager.cc:1875] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 01:59:59.510804 16901 ts_tablet_manager.cc:1888] T 652e8f55cfc245078cac08811d2cfd77 P 12e8c683db054df3b75c6285b98a5ad8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:59.510952 16901 log.cc:1196] T 652e8f55cfc245078cac08811d2cfd77 P 12e8c683db054df3b75c6285b98a5ad8: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/652e8f55cfc245078cac08811d2cfd77
I20240503 01:59:59.511302 16901 ts_tablet_manager.cc:1909] T 652e8f55cfc245078cac08811d2cfd77 P 12e8c683db054df3b75c6285b98a5ad8: Deleting consensus metadata
I20240503 01:59:59.511806 16900 ts_tablet_manager.cc:1888] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 01:59:59.511993 16900 log.cc:1196] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/652e8f55cfc245078cac08811d2cfd77
I20240503 01:59:59.512316 16900 ts_tablet_manager.cc:1909] T 652e8f55cfc245078cac08811d2cfd77 P bb5364d730cb4489a0bfc6c270b7450a: Deleting consensus metadata
I20240503 01:59:59.513599 6727 catalog_manager.cc:4830] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): tablet 652e8f55cfc245078cac08811d2cfd77 (table rebalancer_dynamic_table_new_name [id=9c752edb5c804520b5a2eb92b82e7963]) successfully deleted
I20240503 01:59:59.513820 6725 catalog_manager.cc:4830] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): tablet 652e8f55cfc245078cac08811d2cfd77 (table rebalancer_dynamic_table_new_name [id=9c752edb5c804520b5a2eb92b82e7963]) successfully deleted
I20240503 01:59:59.515821 6881 consensus_queue.cc:237] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 01:59:59.517683 7012 raft_consensus.cc:1261] T c4ac4ad7f9eb4723814f65ce96758c0e P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:59.517640 9363 raft_consensus.cc:1261] T c4ac4ad7f9eb4723814f65ce96758c0e P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:59.518553 16673 consensus_queue.cc:1035] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:59.518879 16401 consensus_queue.cc:1035] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:59.521930 16401 raft_consensus.cc:2932] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:59.522900 9363 raft_consensus.cc:2932] T c4ac4ad7f9eb4723814f65ce96758c0e P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:59.526055 6739 catalog_manager.cc:5484] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:59.528359 7012 raft_consensus.cc:2932] T c4ac4ad7f9eb4723814f65ce96758c0e P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 01:59:59.530200 7125 tablet_service.cc:1514] Processing DeleteTablet for tablet c4ac4ad7f9eb4723814f65ce96758c0e with delete_type TABLET_DATA_TOMBSTONED (TS c688f3df56ac4b6d9b0c30fa4cc8ca2a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:59.531033 16713 tablet_replica.cc:330] T c4ac4ad7f9eb4723814f65ce96758c0e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:59.531231 16713 raft_consensus.cc:2229] T c4ac4ad7f9eb4723814f65ce96758c0e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:59.531410 16713 raft_consensus.cc:2258] T c4ac4ad7f9eb4723814f65ce96758c0e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:59.532065 16713 ts_tablet_manager.cc:1875] T c4ac4ad7f9eb4723814f65ce96758c0e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:59.534269 16713 ts_tablet_manager.cc:1888] T c4ac4ad7f9eb4723814f65ce96758c0e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 01:59:59.534416 16713 log.cc:1196] T c4ac4ad7f9eb4723814f65ce96758c0e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/c4ac4ad7f9eb4723814f65ce96758c0e
I20240503 01:59:59.536952 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet c4ac4ad7f9eb4723814f65ce96758c0e (table rebalance_test_table_19 [id=d2d157b43e4243979ed58dfe4b2fe341]) successfully deleted
I20240503 01:59:59.597270 6734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 01:59:59.604822 9473 tablet_service.cc:1467] Processing CreateTablet for tablet 2f7395b7f19b4c989e1d660945b2522a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=55bd4c107fe440ab9c90f58d467a5e3b]), partition=
I20240503 01:59:59.605072 9605 tablet_service.cc:1467] Processing CreateTablet for tablet 2f7395b7f19b4c989e1d660945b2522a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=55bd4c107fe440ab9c90f58d467a5e3b]), partition=
I20240503 01:59:59.605134 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f7395b7f19b4c989e1d660945b2522a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.605286 9605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f7395b7f19b4c989e1d660945b2522a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.607201 6863 tablet_service.cc:1467] Processing CreateTablet for tablet 2f7395b7f19b4c989e1d660945b2522a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=55bd4c107fe440ab9c90f58d467a5e3b]), partition=
I20240503 01:59:59.607416 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f7395b7f19b4c989e1d660945b2522a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.607617 16953 tablet_bootstrap.cc:492] T 2f7395b7f19b4c989e1d660945b2522a P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:59.608530 16953 tablet_bootstrap.cc:654] T 2f7395b7f19b4c989e1d660945b2522a P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.608804 16952 tablet_bootstrap.cc:492] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:59.609722 16799 tablet_bootstrap.cc:492] T 2f7395b7f19b4c989e1d660945b2522a P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:59.610613 16952 tablet_bootstrap.cc:654] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.610613 16799 tablet_bootstrap.cc:654] T 2f7395b7f19b4c989e1d660945b2522a P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.621218 16953 tablet_bootstrap.cc:492] T 2f7395b7f19b4c989e1d660945b2522a P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:59.621554 16953 ts_tablet_manager.cc:1367] T 2f7395b7f19b4c989e1d660945b2522a P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240503 01:59:59.622113 16953 raft_consensus.cc:348] T 2f7395b7f19b4c989e1d660945b2522a P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:59.622247 16953 raft_consensus.cc:374] T 2f7395b7f19b4c989e1d660945b2522a P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.622290 16953 raft_consensus.cc:729] T 2f7395b7f19b4c989e1d660945b2522a P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.622424 16953 consensus_queue.cc:260] T 2f7395b7f19b4c989e1d660945b2522a P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:59.622889 16953 ts_tablet_manager.cc:1398] T 2f7395b7f19b4c989e1d660945b2522a P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:59.623401 16952 tablet_bootstrap.cc:492] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:59.623401 16799 tablet_bootstrap.cc:492] T 2f7395b7f19b4c989e1d660945b2522a P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:59.623505 16799 ts_tablet_manager.cc:1367] T 2f7395b7f19b4c989e1d660945b2522a P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.001s
I20240503 01:59:59.623620 16952 ts_tablet_manager.cc:1367] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.002s
I20240503 01:59:59.624037 16799 raft_consensus.cc:348] T 2f7395b7f19b4c989e1d660945b2522a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:59.624172 16799 raft_consensus.cc:374] T 2f7395b7f19b4c989e1d660945b2522a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.624217 16799 raft_consensus.cc:729] T 2f7395b7f19b4c989e1d660945b2522a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.624356 16799 consensus_queue.cc:260] T 2f7395b7f19b4c989e1d660945b2522a P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:59.624675 16799 ts_tablet_manager.cc:1398] T 2f7395b7f19b4c989e1d660945b2522a P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:59.626773 16636 consensus_queue.cc:1035] T 168c7491d50c44b39c5698c3f9d2e98f P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:59.641593 14833 consensus_queue.cc:1035] T 168c7491d50c44b39c5698c3f9d2e98f P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:59.645849 16952 raft_consensus.cc:348] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:59.646029 16952 raft_consensus.cc:374] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.646062 16952 raft_consensus.cc:729] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.646185 16952 consensus_queue.cc:260] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:59.646473 16952 ts_tablet_manager.cc:1398] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:59.707065 9342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 01:59:59.709851 9473 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:59.717772 6992 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 01:59:59.718425 6863 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 01:59:59.719192 9605 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:59.728099 9736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 01:59:59.734885 7125 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 01:59:59.738373 7256 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 01:59:59.761940 16828 raft_consensus.cc:482] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:59.762099 16828 raft_consensus.cc:504] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:59.762682 16828 leader_election.cc:288] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:59.766283 9624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7395b7f19b4c989e1d660945b2522a" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 01:59:59.766446 9624 raft_consensus.cc:2454] T 2f7395b7f19b4c989e1d660945b2522a P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:59.767089 9429 leader_election.cc:302] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:59.767789 6737 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 01:59:59.769405 16828 raft_consensus.cc:2781] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:59.769482 16828 raft_consensus.cc:482] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:59.769520 16828 raft_consensus.cc:3037] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.769804 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7395b7f19b4c989e1d660945b2522a" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 01:59:59.769930 6881 raft_consensus.cc:2454] T 2f7395b7f19b4c989e1d660945b2522a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 01:59:59.770772 16828 raft_consensus.cc:504] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:59.771209 16828 leader_election.cc:288] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 01:59:59.771929 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7395b7f19b4c989e1d660945b2522a" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 01:59:59.772053 6881 raft_consensus.cc:3037] T 2f7395b7f19b4c989e1d660945b2522a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.772202 9624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7395b7f19b4c989e1d660945b2522a" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 01:59:59.772323 9624 raft_consensus.cc:3037] T 2f7395b7f19b4c989e1d660945b2522a P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.773202 6881 raft_consensus.cc:2454] T 2f7395b7f19b4c989e1d660945b2522a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:59.773339 9624 raft_consensus.cc:2454] T 2f7395b7f19b4c989e1d660945b2522a P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 01:59:59.773690 9429 leader_election.cc:302] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:59.774153 16828 raft_consensus.cc:2781] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:59.774240 16828 raft_consensus.cc:686] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Becoming Leader. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Running, Role: LEADER
I20240503 01:59:59.774369 16828 consensus_queue.cc:237] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 01:59:59.784332 9342 tablet_service.cc:1467] Processing CreateTablet for tablet d2189e61656a428d8931a82ab94c0198 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=04f8fe42403241498bbc9d67e5a5e903]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:59.784751 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2189e61656a428d8931a82ab94c0198. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.789621 16780 consensus_queue.cc:1035] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:59.796036 7256 tablet_service.cc:1467] Processing CreateTablet for tablet d2189e61656a428d8931a82ab94c0198 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=04f8fe42403241498bbc9d67e5a5e903]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:59.796486 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2189e61656a428d8931a82ab94c0198. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.797324 7125 tablet_service.cc:1467] Processing CreateTablet for tablet 848a183ff5f04ca98a852ccc0ab025ff (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=04f8fe42403241498bbc9d67e5a5e903]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:59.797719 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 848a183ff5f04ca98a852ccc0ab025ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.797868 6737 catalog_manager.cc:5484] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: term changed from 0 to 1, leader changed from <none> to bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 01:59:59.798204 15159 consensus_queue.cc:1035] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:59.799221 7254 tablet_service.cc:1467] Processing CreateTablet for tablet 1eefd7ae55eb4403893f5bd31adc2b4b (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=04f8fe42403241498bbc9d67e5a5e903]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:59.799347 14833 consensus_queue.cc:1035] T 5576de720be6425a80b49499506c363f P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:59.799710 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eefd7ae55eb4403893f5bd31adc2b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.801326 9605 tablet_service.cc:1467] Processing CreateTablet for tablet 848a183ff5f04ca98a852ccc0ab025ff (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=04f8fe42403241498bbc9d67e5a5e903]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:59.801498 6863 tablet_service.cc:1467] Processing CreateTablet for tablet 1eefd7ae55eb4403893f5bd31adc2b4b (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=04f8fe42403241498bbc9d67e5a5e903]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:59.801640 9605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 848a183ff5f04ca98a852ccc0ab025ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.801868 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eefd7ae55eb4403893f5bd31adc2b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.802691 9473 tablet_service.cc:1467] Processing CreateTablet for tablet d2189e61656a428d8931a82ab94c0198 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=04f8fe42403241498bbc9d67e5a5e903]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:59.803007 9473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2189e61656a428d8931a82ab94c0198. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.804277 9474 tablet_service.cc:1467] Processing CreateTablet for tablet 848a183ff5f04ca98a852ccc0ab025ff (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=04f8fe42403241498bbc9d67e5a5e903]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:59.804527 9474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 848a183ff5f04ca98a852ccc0ab025ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.804569 16953 tablet_bootstrap.cc:492] T 848a183ff5f04ca98a852ccc0ab025ff P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 01:59:59.806002 16953 tablet_bootstrap.cc:654] T 848a183ff5f04ca98a852ccc0ab025ff P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.807716 16952 tablet_bootstrap.cc:492] T d2189e61656a428d8931a82ab94c0198 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:59.807725 16799 tablet_bootstrap.cc:492] T 1eefd7ae55eb4403893f5bd31adc2b4b P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 01:59:59.808763 16799 tablet_bootstrap.cc:654] T 1eefd7ae55eb4403893f5bd31adc2b4b P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.808964 16952 tablet_bootstrap.cc:654] T d2189e61656a428d8931a82ab94c0198 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.810117 9472 tablet_service.cc:1467] Processing CreateTablet for tablet 1eefd7ae55eb4403893f5bd31adc2b4b (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=04f8fe42403241498bbc9d67e5a5e903]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 01:59:59.810425 9472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eefd7ae55eb4403893f5bd31adc2b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 01:59:59.815047 16978 tablet_bootstrap.cc:492] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 01:59:59.820514 16975 tablet_bootstrap.cc:492] T d2189e61656a428d8931a82ab94c0198 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 01:59:59.823961 16978 tablet_bootstrap.cc:654] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.824021 16953 tablet_bootstrap.cc:492] T 848a183ff5f04ca98a852ccc0ab025ff P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 01:59:59.824120 16953 ts_tablet_manager.cc:1367] T 848a183ff5f04ca98a852ccc0ab025ff P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.002s
I20240503 01:59:59.824672 16953 raft_consensus.cc:348] T 848a183ff5f04ca98a852ccc0ab025ff P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:59.824796 16953 raft_consensus.cc:374] T 848a183ff5f04ca98a852ccc0ab025ff P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.824823 16953 raft_consensus.cc:729] T 848a183ff5f04ca98a852ccc0ab025ff P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.824934 16953 consensus_queue.cc:260] T 848a183ff5f04ca98a852ccc0ab025ff P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:59.825250 16953 ts_tablet_manager.cc:1398] T 848a183ff5f04ca98a852ccc0ab025ff P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 01:59:59.832533 16975 tablet_bootstrap.cc:654] T d2189e61656a428d8931a82ab94c0198 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.835232 16978 tablet_bootstrap.cc:492] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 01:59:59.842196 16978 ts_tablet_manager.cc:1367] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20240503 01:59:59.842711 16975 tablet_bootstrap.cc:492] T d2189e61656a428d8931a82ab94c0198 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 01:59:59.843286 16975 ts_tablet_manager.cc:1367] T d2189e61656a428d8931a82ab94c0198 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.001s
I20240503 01:59:59.843266 16978 raft_consensus.cc:348] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:59.843418 16978 raft_consensus.cc:374] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.843474 16978 raft_consensus.cc:729] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.843617 16978 consensus_queue.cc:260] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:59.844189 16978 ts_tablet_manager.cc:1398] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 01:59:59.844676 16975 raft_consensus.cc:348] T d2189e61656a428d8931a82ab94c0198 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:59.844871 16975 raft_consensus.cc:374] T d2189e61656a428d8931a82ab94c0198 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.844985 16975 raft_consensus.cc:729] T d2189e61656a428d8931a82ab94c0198 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.845304 16975 consensus_queue.cc:260] T d2189e61656a428d8931a82ab94c0198 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:59.846827 16975 ts_tablet_manager.cc:1398] T d2189e61656a428d8931a82ab94c0198 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 01:59:59.857520 16979 tablet_bootstrap.cc:492] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:59.863021 16952 tablet_bootstrap.cc:492] T d2189e61656a428d8931a82ab94c0198 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:59.863150 16952 ts_tablet_manager.cc:1367] T d2189e61656a428d8931a82ab94c0198 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.055s user 0.002s sys 0.001s
I20240503 01:59:59.863651 16952 raft_consensus.cc:348] T d2189e61656a428d8931a82ab94c0198 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:59.863767 16952 raft_consensus.cc:374] T d2189e61656a428d8931a82ab94c0198 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.863797 16952 raft_consensus.cc:729] T d2189e61656a428d8931a82ab94c0198 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.863910 16952 consensus_queue.cc:260] T d2189e61656a428d8931a82ab94c0198 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:59.864218 16952 ts_tablet_manager.cc:1398] T d2189e61656a428d8931a82ab94c0198 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:59.864410 16952 tablet_bootstrap.cc:492] T 1eefd7ae55eb4403893f5bd31adc2b4b P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:59.865578 16952 tablet_bootstrap.cc:654] T 1eefd7ae55eb4403893f5bd31adc2b4b P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.867058 16636 consensus_queue.cc:1035] T 5576de720be6425a80b49499506c363f P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:59.869805 16952 tablet_bootstrap.cc:492] T 1eefd7ae55eb4403893f5bd31adc2b4b P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:59.871451 16952 ts_tablet_manager.cc:1367] T 1eefd7ae55eb4403893f5bd31adc2b4b P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240503 01:59:59.872128 16952 raft_consensus.cc:348] T 1eefd7ae55eb4403893f5bd31adc2b4b P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:59.872283 16952 raft_consensus.cc:374] T 1eefd7ae55eb4403893f5bd31adc2b4b P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.872344 16952 raft_consensus.cc:729] T 1eefd7ae55eb4403893f5bd31adc2b4b P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.872494 16952 consensus_queue.cc:260] T 1eefd7ae55eb4403893f5bd31adc2b4b P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:59.872881 16952 ts_tablet_manager.cc:1398] T 1eefd7ae55eb4403893f5bd31adc2b4b P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:59.873231 16952 tablet_bootstrap.cc:492] T 848a183ff5f04ca98a852ccc0ab025ff P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 01:59:59.876405 16952 tablet_bootstrap.cc:654] T 848a183ff5f04ca98a852ccc0ab025ff P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.878345 16979 tablet_bootstrap.cc:654] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.878408 16799 tablet_bootstrap.cc:492] T 1eefd7ae55eb4403893f5bd31adc2b4b P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 01:59:59.878551 16799 ts_tablet_manager.cc:1367] T 1eefd7ae55eb4403893f5bd31adc2b4b P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.071s user 0.000s sys 0.002s
I20240503 01:59:59.879110 16952 tablet_bootstrap.cc:492] T 848a183ff5f04ca98a852ccc0ab025ff P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 01:59:59.879117 16799 raft_consensus.cc:348] T 1eefd7ae55eb4403893f5bd31adc2b4b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:59.879206 16952 ts_tablet_manager.cc:1367] T 848a183ff5f04ca98a852ccc0ab025ff P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240503 01:59:59.879241 16799 raft_consensus.cc:374] T 1eefd7ae55eb4403893f5bd31adc2b4b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.879276 16799 raft_consensus.cc:729] T 1eefd7ae55eb4403893f5bd31adc2b4b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.879415 16799 consensus_queue.cc:260] T 1eefd7ae55eb4403893f5bd31adc2b4b P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:59.879644 16952 raft_consensus.cc:348] T 848a183ff5f04ca98a852ccc0ab025ff P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:59.879736 16799 ts_tablet_manager.cc:1398] T 1eefd7ae55eb4403893f5bd31adc2b4b P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 01:59:59.879743 16952 raft_consensus.cc:374] T 848a183ff5f04ca98a852ccc0ab025ff P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.879776 16952 raft_consensus.cc:729] T 848a183ff5f04ca98a852ccc0ab025ff P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.879890 16952 consensus_queue.cc:260] T 848a183ff5f04ca98a852ccc0ab025ff P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:59.880160 16952 ts_tablet_manager.cc:1398] T 848a183ff5f04ca98a852ccc0ab025ff P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 01:59:59.883570 16979 tablet_bootstrap.cc:492] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:59.883903 16979 ts_tablet_manager.cc:1367] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.001s
I20240503 01:59:59.884682 16979 raft_consensus.cc:348] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:59.884922 16979 raft_consensus.cc:374] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.884958 16979 raft_consensus.cc:729] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.885099 16979 consensus_queue.cc:260] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 01:59:59.886108 16979 ts_tablet_manager.cc:1398] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 01:59:59.886628 16979 tablet_bootstrap.cc:492] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 01:59:59.887758 16979 tablet_bootstrap.cc:654] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 01:59:59.889098 16979 tablet_bootstrap.cc:492] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 01:59:59.889200 16979 ts_tablet_manager.cc:1367] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 01:59:59.889629 16979 raft_consensus.cc:348] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:59.889735 16979 raft_consensus.cc:374] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 01:59:59.889770 16979 raft_consensus.cc:729] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 01:59:59.889914 16979 consensus_queue.cc:260] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:59.890246 16979 ts_tablet_manager.cc:1398] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 01:59:59.906395 16782 raft_consensus.cc:482] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 01:59:59.906562 16782 raft_consensus.cc:504] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:59.907191 16782 leader_election.cc:288] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:59.907567 9361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2189e61656a428d8931a82ab94c0198" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 01:59:59.907725 9361 raft_consensus.cc:2454] T d2189e61656a428d8931a82ab94c0198 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:59.908041 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2189e61656a428d8931a82ab94c0198" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 01:59:59.908073 7211 leader_election.cc:302] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 1d0e738c34af40fa9950c6f086619b3a; no voters:
I20240503 01:59:59.908187 9494 raft_consensus.cc:2454] T d2189e61656a428d8931a82ab94c0198 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 01:59:59.908236 16782 raft_consensus.cc:2781] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 01:59:59.908282 16782 raft_consensus.cc:482] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 01:59:59.908321 16782 raft_consensus.cc:3037] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.909492 16782 raft_consensus.cc:504] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:59.909895 16782 leader_election.cc:288] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Requested vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 01:59:59.910611 9361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2189e61656a428d8931a82ab94c0198" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 01:59:59.910740 9361 raft_consensus.cc:3037] T d2189e61656a428d8931a82ab94c0198 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.911058 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2189e61656a428d8931a82ab94c0198" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 01:59:59.911195 9494 raft_consensus.cc:3037] T d2189e61656a428d8931a82ab94c0198 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 01:59:59.911774 9361 raft_consensus.cc:2454] T d2189e61656a428d8931a82ab94c0198 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:59.912057 9494 raft_consensus.cc:2454] T d2189e61656a428d8931a82ab94c0198 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 01:59:59.912425 7209 leader_election.cc:302] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 01:59:59.912595 16782 raft_consensus.cc:2781] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 01:59:59.912662 16782 raft_consensus.cc:686] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 01:59:59.912796 16782 consensus_queue.cc:237] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 01:59:59.916579 6737 catalog_manager.cc:5484] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 01:59:59.922948 6881 consensus_queue.cc:237] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 01:59:59.924582 7274 raft_consensus.cc:1261] T 4bd8226227c348d5a5d4d8ff28959d27 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:59.924993 9624 raft_consensus.cc:1261] T 4bd8226227c348d5a5d4d8ff28959d27 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 01:59:59.925081 16910 consensus_queue.cc:1035] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:59.925423 16673 consensus_queue.cc:1035] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 01:59:59.927313 16910 raft_consensus.cc:2932] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:59.928237 9624 raft_consensus.cc:2932] T 4bd8226227c348d5a5d4d8ff28959d27 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:59.929355 7274 raft_consensus.cc:2932] T 4bd8226227c348d5a5d4d8ff28959d27 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:59.931581 6737 catalog_manager.cc:5484] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index 4 to 5, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 01:59:59.935828 7125 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bd8226227c348d5a5d4d8ff28959d27 with delete_type TABLET_DATA_TOMBSTONED (TS c688f3df56ac4b6d9b0c30fa4cc8ca2a not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50370
I20240503 01:59:59.936081 16713 tablet_replica.cc:330] T 4bd8226227c348d5a5d4d8ff28959d27 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 01:59:59.936251 16713 raft_consensus.cc:2229] T 4bd8226227c348d5a5d4d8ff28959d27 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 01:59:59.936400 16713 raft_consensus.cc:2258] T 4bd8226227c348d5a5d4d8ff28959d27 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 01:59:59.936928 16713 ts_tablet_manager.cc:1875] T 4bd8226227c348d5a5d4d8ff28959d27 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 01:59:59.940235 16713 ts_tablet_manager.cc:1888] T 4bd8226227c348d5a5d4d8ff28959d27 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 01:59:59.940389 16713 log.cc:1196] T 4bd8226227c348d5a5d4d8ff28959d27 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/4bd8226227c348d5a5d4d8ff28959d27
I20240503 01:59:59.941174 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet 4bd8226227c348d5a5d4d8ff28959d27 (table rebalance_test_table_3 [id=45e94c90bda847688db4572a4f9d0c02]) successfully deleted
I20240503 01:59:59.942054 6741 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 01:59:59.948033 6881 raft_consensus.cc:1261] T 2f7395b7f19b4c989e1d660945b2522a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:59.948225 9624 raft_consensus.cc:1261] T 2f7395b7f19b4c989e1d660945b2522a P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 01:59:59.948531 16828 consensus_queue.cc:1035] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:59.948662 16819 consensus_queue.cc:1035] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 01:59:59.952121 6741 catalog_manager.cc:5576] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943) does not have the latest schema for tablet 2f7395b7f19b4c989e1d660945b2522a (table rebalancer_dynamic_table [id=55bd4c107fe440ab9c90f58d467a5e3b]). Expected version 1 got 0
I20240503 01:59:59.952847 6737 catalog_manager.cc:5576] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087) does not have the latest schema for tablet 2f7395b7f19b4c989e1d660945b2522a (table rebalancer_dynamic_table [id=55bd4c107fe440ab9c90f58d467a5e3b]). Expected version 1 got 0
I20240503 01:59:59.954138 17028 tablet.cc:1722] T 2f7395b7f19b4c989e1d660945b2522a P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:59.954718 17028 tablet.cc:1620] T 2f7395b7f19b4c989e1d660945b2522a P 5dcc33cc621a4cbbae155f3230fb7c46: MemRowSet was empty: no flush needed.
I20240503 01:59:59.960520 6737 catalog_manager.cc:5576] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087) does not have the latest schema for tablet 2f7395b7f19b4c989e1d660945b2522a (table rebalancer_dynamic_table [id=55bd4c107fe440ab9c90f58d467a5e3b]). Expected version 1 got 0
I20240503 01:59:59.960745 6741 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=55bd4c107fe440ab9c90f58d467a5e3b] alter complete (version 1)
I20240503 01:59:59.962316 17027 tablet.cc:1722] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:59.962744 17027 tablet.cc:1620] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 01:59:59.965554 17029 tablet.cc:1722] T 2f7395b7f19b4c989e1d660945b2522a P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 01:59:59.968097 17029 tablet.cc:1620] T 2f7395b7f19b4c989e1d660945b2522a P b7062fbbc2b54fff9e99ec4ea5562c76: MemRowSet was empty: no flush needed.
I20240503 02:00:00.062909 6738 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:00.069319 17027 tablet.cc:1722] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:00.069705 17027 tablet.cc:1620] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 02:00:00.070125 17028 tablet.cc:1722] T 2f7395b7f19b4c989e1d660945b2522a P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:00.070447 17028 tablet.cc:1620] T 2f7395b7f19b4c989e1d660945b2522a P 5dcc33cc621a4cbbae155f3230fb7c46: MemRowSet was empty: no flush needed.
I20240503 02:00:00.070947 17029 tablet.cc:1722] T 2f7395b7f19b4c989e1d660945b2522a P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:00.071251 17029 tablet.cc:1620] T 2f7395b7f19b4c989e1d660945b2522a P b7062fbbc2b54fff9e99ec4ea5562c76: MemRowSet was empty: no flush needed.
I20240503 02:00:00.085959 6741 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=55bd4c107fe440ab9c90f58d467a5e3b] alter complete (version 2)
I20240503 02:00:00.097474 16597 raft_consensus.cc:482] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:00.097682 16597 raft_consensus.cc:504] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:00.098313 16597 leader_election.cc:288] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 02:00:00.099040 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eefd7ae55eb4403893f5bd31adc2b4b" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 02:00:00.099258 9494 raft_consensus.cc:2454] T 1eefd7ae55eb4403893f5bd31adc2b4b P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 02:00:00.099251 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eefd7ae55eb4403893f5bd31adc2b4b" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 02:00:00.099373 6881 raft_consensus.cc:2454] T 1eefd7ae55eb4403893f5bd31adc2b4b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 02:00:00.099769 7209 leader_election.cc:302] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 02:00:00.100164 16597 raft_consensus.cc:2781] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:00.100229 16597 raft_consensus.cc:482] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:00.100270 16597 raft_consensus.cc:3037] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.102095 16597 raft_consensus.cc:504] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:00.102540 16597 leader_election.cc:288] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Requested vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 02:00:00.103175 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eefd7ae55eb4403893f5bd31adc2b4b" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 02:00:00.103204 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eefd7ae55eb4403893f5bd31adc2b4b" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 02:00:00.103340 6881 raft_consensus.cc:3037] T 1eefd7ae55eb4403893f5bd31adc2b4b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.103341 9494 raft_consensus.cc:3037] T 1eefd7ae55eb4403893f5bd31adc2b4b P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.104791 9494 raft_consensus.cc:2454] T 1eefd7ae55eb4403893f5bd31adc2b4b P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 02:00:00.104791 6881 raft_consensus.cc:2454] T 1eefd7ae55eb4403893f5bd31adc2b4b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 02:00:00.105432 7212 leader_election.cc:302] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 02:00:00.106491 16597 raft_consensus.cc:2781] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:00.106621 16597 raft_consensus.cc:686] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 02:00:00.106850 16597 consensus_queue.cc:237] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:00.118726 6735 catalog_manager.cc:5484] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:00.149860 15159 raft_consensus.cc:482] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:00.150023 15159 raft_consensus.cc:504] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:00.150460 15159 leader_election.cc:288] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:00.150766 9624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "848a183ff5f04ca98a852ccc0ab025ff" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 02:00:00.150911 9624 raft_consensus.cc:2454] T 848a183ff5f04ca98a852ccc0ab025ff P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 02:00:00.151129 7080 leader_election.cc:302] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 02:00:00.151096 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "848a183ff5f04ca98a852ccc0ab025ff" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 02:00:00.151237 9493 raft_consensus.cc:2454] T 848a183ff5f04ca98a852ccc0ab025ff P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 02:00:00.151786 15159 raft_consensus.cc:2781] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:00.151860 15159 raft_consensus.cc:482] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:00.151894 15159 raft_consensus.cc:3037] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.152988 15159 raft_consensus.cc:504] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:00.153373 15159 leader_election.cc:288] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Requested vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:00.155901 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "848a183ff5f04ca98a852ccc0ab025ff" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 02:00:00.156111 9494 raft_consensus.cc:3037] T 848a183ff5f04ca98a852ccc0ab025ff P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.157356 9494 raft_consensus.cc:2454] T 848a183ff5f04ca98a852ccc0ab025ff P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 02:00:00.159754 7078 leader_election.cc:302] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 02:00:00.161770 9624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "848a183ff5f04ca98a852ccc0ab025ff" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 02:00:00.161936 9624 raft_consensus.cc:3037] T 848a183ff5f04ca98a852ccc0ab025ff P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.163166 9624 raft_consensus.cc:2454] T 848a183ff5f04ca98a852ccc0ab025ff P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 02:00:00.163761 15159 raft_consensus.cc:2781] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:00.163882 15159 raft_consensus.cc:686] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Becoming Leader. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Running, Role: LEADER
I20240503 02:00:00.164034 15159 consensus_queue.cc:237] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:00.169553 6737 catalog_manager.cc:5484] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:00.189002 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:00.198407 17027 tablet.cc:1722] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:00.199040 17029 tablet.cc:1722] T 2f7395b7f19b4c989e1d660945b2522a P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:00.203155 17028 tablet.cc:1722] T 2f7395b7f19b4c989e1d660945b2522a P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:00.206672 6738 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=55bd4c107fe440ab9c90f58d467a5e3b] alter complete (version 3)
I20240503 02:00:00.229488 9342 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:00:00.237015 6863 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:00.240198 6992 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:00:00.254889 9474 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:00:00.255184 9605 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:00:00.283672 7256 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:00.283716 9736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:00:00.302191 7125 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:00:00.311791 6734 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:00.312045 6734 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:00.315914 6734 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 2f7395b7f19b4c989e1d660945b2522a
I20240503 02:00:00.316849 9605 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f7395b7f19b4c989e1d660945b2522a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:00 UTC) from {username='slave'} at 127.0.0.1:58562
I20240503 02:00:00.317239 17095 tablet_replica.cc:330] T 2f7395b7f19b4c989e1d660945b2522a P b7062fbbc2b54fff9e99ec4ea5562c76: stopping tablet replica
I20240503 02:00:00.317229 9474 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f7395b7f19b4c989e1d660945b2522a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:00 UTC) from {username='slave'} at 127.0.0.1:54678
I20240503 02:00:00.317531 17095 raft_consensus.cc:2229] T 2f7395b7f19b4c989e1d660945b2522a P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:00.317739 17095 raft_consensus.cc:2258] T 2f7395b7f19b4c989e1d660945b2522a P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:00.317781 6863 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f7395b7f19b4c989e1d660945b2522a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:00 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 02:00:00.318107 17097 tablet_replica.cc:330] T 2f7395b7f19b4c989e1d660945b2522a P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 02:00:00.318239 17097 raft_consensus.cc:2229] T 2f7395b7f19b4c989e1d660945b2522a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:00.318382 17097 raft_consensus.cc:2258] T 2f7395b7f19b4c989e1d660945b2522a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:00.319108 17097 ts_tablet_manager.cc:1875] T 2f7395b7f19b4c989e1d660945b2522a P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:00.319572 17095 ts_tablet_manager.cc:1875] T 2f7395b7f19b4c989e1d660945b2522a P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:00.322032 17097 ts_tablet_manager.cc:1888] T 2f7395b7f19b4c989e1d660945b2522a P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:00.322116 17097 log.cc:1196] T 2f7395b7f19b4c989e1d660945b2522a P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/2f7395b7f19b4c989e1d660945b2522a
I20240503 02:00:00.322420 17097 ts_tablet_manager.cc:1909] T 2f7395b7f19b4c989e1d660945b2522a P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 02:00:00.322677 17096 tablet_replica.cc:330] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a: stopping tablet replica
I20240503 02:00:00.322875 17096 raft_consensus.cc:2229] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:00.323132 17096 raft_consensus.cc:2258] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:00.323531 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 2f7395b7f19b4c989e1d660945b2522a (table rebalancer_dynamic_table_new_name [id=55bd4c107fe440ab9c90f58d467a5e3b]) successfully deleted
I20240503 02:00:00.323704 17096 ts_tablet_manager.cc:1875] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:00.327224 17096 ts_tablet_manager.cc:1888] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:00.327402 17096 log.cc:1196] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/2f7395b7f19b4c989e1d660945b2522a
I20240503 02:00:00.327752 17096 ts_tablet_manager.cc:1909] T 2f7395b7f19b4c989e1d660945b2522a P bb5364d730cb4489a0bfc6c270b7450a: Deleting consensus metadata
I20240503 02:00:00.328557 6725 catalog_manager.cc:4830] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): tablet 2f7395b7f19b4c989e1d660945b2522a (table rebalancer_dynamic_table_new_name [id=55bd4c107fe440ab9c90f58d467a5e3b]) successfully deleted
I20240503 02:00:00.333408 17095 ts_tablet_manager.cc:1888] T 2f7395b7f19b4c989e1d660945b2522a P b7062fbbc2b54fff9e99ec4ea5562c76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:00.333789 17095 log.cc:1196] T 2f7395b7f19b4c989e1d660945b2522a P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/2f7395b7f19b4c989e1d660945b2522a
I20240503 02:00:00.336388 17095 ts_tablet_manager.cc:1909] T 2f7395b7f19b4c989e1d660945b2522a P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting consensus metadata
I20240503 02:00:00.338734 6727 catalog_manager.cc:4830] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): tablet 2f7395b7f19b4c989e1d660945b2522a (table rebalancer_dynamic_table_new_name [id=55bd4c107fe440ab9c90f58d467a5e3b]) successfully deleted
I20240503 02:00:00.367238 16597 consensus_queue.cc:1035] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:00.374430 16782 consensus_queue.cc:1035] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:00.417905 7145 consensus_queue.cc:237] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:00.418597 6734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20240503 02:00:00.419746 7079 consensus_peers.cc:485] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): Couldn't send request to peer db1a0000aaa346f79c0aa4b6c5105191. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ca4e33409b74f359d3242248652d8bb. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:00.420198 7274 raft_consensus.cc:1261] T 8ca4e33409b74f359d3242248652d8bb P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:00.420228 9625 raft_consensus.cc:1261] T 8ca4e33409b74f359d3242248652d8bb P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:00.420636 17118 consensus_queue.cc:1035] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:00.420858 16780 consensus_queue.cc:1035] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:00.425773 16780 raft_consensus.cc:2932] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:00.430370 9342 tablet_service.cc:1467] Processing CreateTablet for tablet fd8202c2dfbe4a24b8b6324ca3e31cf1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=090454629ddd48d0b5d8047a6b6d6976]), partition=
I20240503 02:00:00.430730 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd8202c2dfbe4a24b8b6324ca3e31cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.431208 9736 tablet_service.cc:1467] Processing CreateTablet for tablet fd8202c2dfbe4a24b8b6324ca3e31cf1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=090454629ddd48d0b5d8047a6b6d6976]), partition=
I20240503 02:00:00.431423 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd8202c2dfbe4a24b8b6324ca3e31cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.433427 17122 tablet_bootstrap.cc:492] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 02:00:00.434494 7125 tablet_service.cc:1467] Processing CreateTablet for tablet fd8202c2dfbe4a24b8b6324ca3e31cf1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=090454629ddd48d0b5d8047a6b6d6976]), partition=
I20240503 02:00:00.435149 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd8202c2dfbe4a24b8b6324ca3e31cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.435672 17122 tablet_bootstrap.cc:654] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:00.436393 9625 raft_consensus.cc:2932] T 8ca4e33409b74f359d3242248652d8bb P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:00.437044 6734 catalog_manager.cc:5484] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index -1 to 2, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:00.438148 17123 tablet_bootstrap.cc:492] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 02:00:00.439195 17123 tablet_bootstrap.cc:654] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:00.440009 17122 tablet_bootstrap.cc:492] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 02:00:00.440145 17122 ts_tablet_manager.cc:1367] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:00.440732 17122 raft_consensus.cc:348] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:00.440857 17122 raft_consensus.cc:374] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:00.440891 17122 raft_consensus.cc:729] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 02:00:00.441036 17122 consensus_queue.cc:260] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:00.441394 17122 ts_tablet_manager.cc:1398] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:00.441488 17123 tablet_bootstrap.cc:492] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 02:00:00.441577 17123 ts_tablet_manager.cc:1367] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:00.442087 17123 raft_consensus.cc:348] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:00.442191 17123 raft_consensus.cc:374] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:00.442221 17123 raft_consensus.cc:729] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 02:00:00.442335 17123 consensus_queue.cc:260] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:00.444089 17123 ts_tablet_manager.cc:1398] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:00.444443 7274 raft_consensus.cc:2932] T 8ca4e33409b74f359d3242248652d8bb P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:00.452603 17125 tablet_bootstrap.cc:492] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 02:00:00.454350 17125 tablet_bootstrap.cc:654] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:00.455878 17125 tablet_bootstrap.cc:492] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 02:00:00.456032 17125 ts_tablet_manager.cc:1367] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:00.456658 17125 raft_consensus.cc:348] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:00.456826 17125 raft_consensus.cc:374] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:00.456892 17125 raft_consensus.cc:729] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 02:00:00.457227 17125 consensus_queue.cc:260] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:00.457648 17125 ts_tablet_manager.cc:1398] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:00.472397 9361 consensus_queue.cc:237] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:00.474184 9624 raft_consensus.cc:1261] T 9b8976486acf49d5a9d0d496379ee2a6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:00.474349 9494 raft_consensus.cc:1261] T 9b8976486acf49d5a9d0d496379ee2a6 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:00.474387 9299 consensus_peers.cc:485] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8 -> Peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): Couldn't send request to peer 5dcc33cc621a4cbbae155f3230fb7c46. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b8976486acf49d5a9d0d496379ee2a6. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:00.474875 16988 consensus_queue.cc:1035] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:00.475266 16988 consensus_queue.cc:1035] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:00.478191 16988 raft_consensus.cc:2932] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:00.478678 9494 raft_consensus.cc:2932] T 9b8976486acf49d5a9d0d496379ee2a6 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:00.482861 9624 raft_consensus.cc:2932] T 9b8976486acf49d5a9d0d496379ee2a6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:00.483486 6735 catalog_manager.cc:5484] T 9b8976486acf49d5a9d0d496379ee2a6 P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:00.510972 9361 consensus_queue.cc:237] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:00.512919 9757 raft_consensus.cc:1261] T 6fd0378b2cff42a681361930f487d834 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:00.513125 9296 consensus_peers.cc:485] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 -> Peer 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): Couldn't send request to peer 63331f455be049c1bdd6c23c59ac1bef. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6fd0378b2cff42a681361930f487d834. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:00.513545 16988 consensus_queue.cc:1035] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:00.514261 9494 raft_consensus.cc:1261] T 6fd0378b2cff42a681361930f487d834 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:00.514988 16988 consensus_queue.cc:1035] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:00.517072 16988 raft_consensus.cc:2932] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:00.517446 9494 raft_consensus.cc:2932] T 6fd0378b2cff42a681361930f487d834 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:00.521104 9757 raft_consensus.cc:2932] T 6fd0378b2cff42a681361930f487d834 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:00.521375 6741 catalog_manager.cc:5484] T 6fd0378b2cff42a681361930f487d834 P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:00.558099 9757 consensus_queue.cc:237] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:00.559999 7274 raft_consensus.cc:1261] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:00.560356 9361 raft_consensus.cc:1261] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:00.560911 9692 consensus_peers.cc:485] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): Couldn't send request to peer c688f3df56ac4b6d9b0c30fa4cc8ca2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0bbbd17c78bc4e329a3a7c7008a34d86. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:00.561169 17183 consensus_queue.cc:1035] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:00.561949 16785 consensus_queue.cc:1035] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:00.565235 17183 raft_consensus.cc:2932] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:00.565649 9361 raft_consensus.cc:2932] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:00.567821 7274 raft_consensus.cc:2932] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:00.569133 6735 catalog_manager.cc:5484] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:00.593672 17144 raft_consensus.cc:482] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:00.593818 17144 raft_consensus.cc:504] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:00.594224 17144 leader_election.cc:288] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 02:00:00.594563 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd8202c2dfbe4a24b8b6324ca3e31cf1" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 02:00:00.594583 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd8202c2dfbe4a24b8b6324ca3e31cf1" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 02:00:00.594703 9757 raft_consensus.cc:2454] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 02:00:00.594729 7145 raft_consensus.cc:2454] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 02:00:00.595032 9297 leader_election.cc:302] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 02:00:00.595263 17144 raft_consensus.cc:2781] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:00.595336 17144 raft_consensus.cc:482] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:00.595381 17144 raft_consensus.cc:3037] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.596474 17144 raft_consensus.cc:504] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:00.596915 17144 leader_election.cc:288] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Requested vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 02:00:00.597411 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd8202c2dfbe4a24b8b6324ca3e31cf1" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 02:00:00.597564 9757 raft_consensus.cc:3037] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.597877 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd8202c2dfbe4a24b8b6324ca3e31cf1" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 02:00:00.598030 7145 raft_consensus.cc:3037] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.598488 6881 consensus_queue.cc:237] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:00.598762 9757 raft_consensus.cc:2454] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 02:00:00.599107 7145 raft_consensus.cc:2454] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 02:00:00.599110 9297 leader_election.cc:302] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 02:00:00.599534 17144 raft_consensus.cc:2781] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:00.599638 17144 raft_consensus.cc:686] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Running, Role: LEADER
I20240503 02:00:00.599779 17144 consensus_queue.cc:237] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:00.600186 9624 raft_consensus.cc:1261] T 4bd8226227c348d5a5d4d8ff28959d27 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:00.600620 7274 raft_consensus.cc:1261] T 4bd8226227c348d5a5d4d8ff28959d27 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:00.601055 16401 consensus_queue.cc:1035] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 02:00:00.601346 6817 consensus_peers.cc:485] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): Couldn't send request to peer db1a0000aaa346f79c0aa4b6c5105191. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4bd8226227c348d5a5d4d8ff28959d27. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:00.601445 16401 consensus_queue.cc:1035] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:00.603837 6734 catalog_manager.cc:5484] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:00.606199 16673 raft_consensus.cc:2932] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:00.606760 7274 raft_consensus.cc:2932] T 4bd8226227c348d5a5d4d8ff28959d27 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:00.610015 9624 raft_consensus.cc:2932] T 4bd8226227c348d5a5d4d8ff28959d27 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:00.610509 6734 catalog_manager.cc:5484] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index 5 to 6, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:00.610833 16780 consensus_queue.cc:1035] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:00.617897 15159 consensus_queue.cc:1035] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:00.649555 9494 consensus_queue.cc:237] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:00.651363 9361 raft_consensus.cc:1261] T 5089809a9f3846f5b30397b1062d4e55 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:00.651463 9624 raft_consensus.cc:1261] T 5089809a9f3846f5b30397b1062d4e55 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:00.651692 9430 consensus_peers.cc:485] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a -> Peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): Couldn't send request to peer 5dcc33cc621a4cbbae155f3230fb7c46. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5089809a9f3846f5b30397b1062d4e55. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:00.651868 16828 consensus_queue.cc:1035] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:00.652292 16819 consensus_queue.cc:1035] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:00.655162 16819 raft_consensus.cc:2932] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:00.655778 9624 raft_consensus.cc:2932] T 5089809a9f3846f5b30397b1062d4e55 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:00.658893 9363 raft_consensus.cc:2932] T 5089809a9f3846f5b30397b1062d4e55 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:00.659365 6741 catalog_manager.cc:5484] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:00.685688 9625 consensus_queue.cc:237] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:00.687705 9494 raft_consensus.cc:1261] T b93fe374d2a9496a847480b7824a5937 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer b7062fbbc2b54fff9e99ec4ea5562c76: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:00.687738 9757 raft_consensus.cc:1261] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer b7062fbbc2b54fff9e99ec4ea5562c76: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:00.687856 9559 consensus_peers.cc:485] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76 -> Peer 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): Couldn't send request to peer 63331f455be049c1bdd6c23c59ac1bef. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b93fe374d2a9496a847480b7824a5937. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:00.688098 17241 consensus_queue.cc:1035] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:00.688321 16102 consensus_queue.cc:1035] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:00.690306 16102 raft_consensus.cc:2932] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:00.691071 9494 raft_consensus.cc:2932] T b93fe374d2a9496a847480b7824a5937 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:00.692972 9757 raft_consensus.cc:2932] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:00.694175 6734 catalog_manager.cc:5484] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76 reported cstate change: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:00.714162 16782 consensus_queue.cc:1035] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:00.717964 16597 consensus_queue.cc:1035] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:00.729182 9755 consensus_queue.cc:237] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:00.730329 6735 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:00.731725 6881 raft_consensus.cc:1261] T 9e21d082e40540d4bb908fc9e90055c9 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:00.731912 9692 consensus_peers.cc:485] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): Couldn't send request to peer c688f3df56ac4b6d9b0c30fa4cc8ca2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e21d082e40540d4bb908fc9e90055c9. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:00.732179 9363 raft_consensus.cc:1261] T 9e21d082e40540d4bb908fc9e90055c9 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:00.732769 6741 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:00.732918 17183 consensus_queue.cc:1035] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:00.734190 17183 consensus_queue.cc:1035] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:00.736604 17183 raft_consensus.cc:2932] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:00.740036 9363 raft_consensus.cc:2932] T 9e21d082e40540d4bb908fc9e90055c9 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:00.740370 7145 raft_consensus.cc:1261] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:00.741247 9474 tablet_service.cc:1467] Processing CreateTablet for tablet 5a1b7f0005e54894b0bc28c3b988a14a (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=995a6b1d9ef94c0dbb5d2604eedda4ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:00.741559 9474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a1b7f0005e54894b0bc28c3b988a14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.742221 9755 raft_consensus.cc:1261] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:00.742408 6992 tablet_service.cc:1467] Processing CreateTablet for tablet 9d44274085b9441c9f15676d842a0710 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=995a6b1d9ef94c0dbb5d2604eedda4ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:00.742790 9472 tablet_service.cc:1467] Processing CreateTablet for tablet 90600b5313d64a10bf00fc1effd176e6 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=995a6b1d9ef94c0dbb5d2604eedda4ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:00.742709 6881 raft_consensus.cc:2932] T 9e21d082e40540d4bb908fc9e90055c9 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:00.742910 15917 consensus_queue.cc:1035] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:00.743050 9472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90600b5313d64a10bf00fc1effd176e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.743813 6992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d44274085b9441c9f15676d842a0710. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.748410 6734 catalog_manager.cc:5484] T 9e21d082e40540d4bb908fc9e90055c9 P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:00.749020 7124 tablet_service.cc:1467] Processing CreateTablet for tablet 90600b5313d64a10bf00fc1effd176e6 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=995a6b1d9ef94c0dbb5d2604eedda4ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:00.749431 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90600b5313d64a10bf00fc1effd176e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.750247 7125 tablet_service.cc:1467] Processing CreateTablet for tablet 5a1b7f0005e54894b0bc28c3b988a14a (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=995a6b1d9ef94c0dbb5d2604eedda4ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:00.750591 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a1b7f0005e54894b0bc28c3b988a14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.752312 17274 tablet.cc:1722] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:00.752740 17274 tablet.cc:1620] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 02:00:00.753618 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 9d44274085b9441c9f15676d842a0710 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=995a6b1d9ef94c0dbb5d2604eedda4ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:00.753929 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d44274085b9441c9f15676d842a0710. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.755112 9605 tablet_service.cc:1467] Processing CreateTablet for tablet 5a1b7f0005e54894b0bc28c3b988a14a (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=995a6b1d9ef94c0dbb5d2604eedda4ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:00.755383 9605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a1b7f0005e54894b0bc28c3b988a14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.755322 17144 consensus_queue.cc:1035] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:00.756752 9736 tablet_service.cc:1467] Processing CreateTablet for tablet 9d44274085b9441c9f15676d842a0710 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=995a6b1d9ef94c0dbb5d2604eedda4ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:00.756937 17125 tablet_bootstrap.cc:492] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 02:00:00.757036 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d44274085b9441c9f15676d842a0710. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.757846 9604 tablet_service.cc:1467] Processing CreateTablet for tablet 90600b5313d64a10bf00fc1effd176e6 (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=995a6b1d9ef94c0dbb5d2604eedda4ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:00.758013 17125 tablet_bootstrap.cc:654] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:00.758065 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90600b5313d64a10bf00fc1effd176e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.759776 17122 tablet_bootstrap.cc:492] T 9d44274085b9441c9f15676d842a0710 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 02:00:00.760591 17278 tablet.cc:1722] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:00.760838 17122 tablet_bootstrap.cc:654] T 9d44274085b9441c9f15676d842a0710 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:00.760869 17278 tablet.cc:1620] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: MemRowSet was empty: no flush needed.
I20240503 02:00:00.762239 17123 tablet_bootstrap.cc:492] T 9d44274085b9441c9f15676d842a0710 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 02:00:00.763226 17123 tablet_bootstrap.cc:654] T 9d44274085b9441c9f15676d842a0710 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:00.766434 17123 tablet_bootstrap.cc:492] T 9d44274085b9441c9f15676d842a0710 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 02:00:00.766571 17123 ts_tablet_manager.cc:1367] T 9d44274085b9441c9f15676d842a0710 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:00.767102 17123 raft_consensus.cc:348] T 9d44274085b9441c9f15676d842a0710 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:00.767189 17123 raft_consensus.cc:374] T 9d44274085b9441c9f15676d842a0710 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:00.767210 17123 raft_consensus.cc:729] T 9d44274085b9441c9f15676d842a0710 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 02:00:00.767396 17123 consensus_queue.cc:260] T 9d44274085b9441c9f15676d842a0710 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:00.767750 17123 ts_tablet_manager.cc:1398] T 9d44274085b9441c9f15676d842a0710 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:00.768034 17269 tablet_bootstrap.cc:492] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 02:00:00.769202 17269 tablet_bootstrap.cc:654] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:00.772200 6740 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=090454629ddd48d0b5d8047a6b6d6976] alter complete (version 1)
I20240503 02:00:00.773779 17276 tablet_bootstrap.cc:492] T 5a1b7f0005e54894b0bc28c3b988a14a P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 02:00:00.773797 17125 tablet_bootstrap.cc:492] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 02:00:00.773891 17125 ts_tablet_manager.cc:1367] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.001s
I20240503 02:00:00.773988 6741 catalog_manager.cc:5576] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053) does not have the latest schema for tablet fd8202c2dfbe4a24b8b6324ca3e31cf1 (table rebalancer_dynamic_table [id=090454629ddd48d0b5d8047a6b6d6976]). Expected version 1 got 0
I20240503 02:00:00.774116 17281 tablet_bootstrap.cc:492] T 5a1b7f0005e54894b0bc28c3b988a14a P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 02:00:00.774392 17125 raft_consensus.cc:348] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:00.774500 17125 raft_consensus.cc:374] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:00.774950 17125 raft_consensus.cc:729] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 02:00:00.775100 17125 consensus_queue.cc:260] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:00.775405 17125 ts_tablet_manager.cc:1398] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:00.775813 17125 tablet_bootstrap.cc:492] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 02:00:00.776014 17283 tablet.cc:1722] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:00.776316 17283 tablet.cc:1620] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 02:00:00.776362 17122 tablet_bootstrap.cc:492] T 9d44274085b9441c9f15676d842a0710 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 02:00:00.776458 17122 ts_tablet_manager.cc:1367] T 9d44274085b9441c9f15676d842a0710 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 02:00:00.776844 17125 tablet_bootstrap.cc:654] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:00.776985 17122 raft_consensus.cc:348] T 9d44274085b9441c9f15676d842a0710 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:00.777101 17122 raft_consensus.cc:374] T 9d44274085b9441c9f15676d842a0710 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:00.777132 17122 raft_consensus.cc:729] T 9d44274085b9441c9f15676d842a0710 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 02:00:00.777261 17122 consensus_queue.cc:260] T 9d44274085b9441c9f15676d842a0710 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:00.777591 17122 ts_tablet_manager.cc:1398] T 9d44274085b9441c9f15676d842a0710 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:00.779450 17125 tablet_bootstrap.cc:492] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 02:00:00.779557 17125 ts_tablet_manager.cc:1367] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:00:00.780313 17125 raft_consensus.cc:348] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:00.780443 17125 raft_consensus.cc:374] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:00.780483 17125 raft_consensus.cc:729] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 02:00:00.780627 17125 consensus_queue.cc:260] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:00.780988 17125 ts_tablet_manager.cc:1398] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 02:00:00.786775 17281 tablet_bootstrap.cc:654] T 5a1b7f0005e54894b0bc28c3b988a14a P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:00.787331 17276 tablet_bootstrap.cc:654] T 5a1b7f0005e54894b0bc28c3b988a14a P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:00.788805 17281 tablet_bootstrap.cc:492] T 5a1b7f0005e54894b0bc28c3b988a14a P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 02:00:00.788978 17281 ts_tablet_manager.cc:1367] T 5a1b7f0005e54894b0bc28c3b988a14a P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 02:00:00.789376 17269 tablet_bootstrap.cc:492] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 02:00:00.789484 17269 ts_tablet_manager.cc:1367] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240503 02:00:00.789721 17281 raft_consensus.cc:348] T 5a1b7f0005e54894b0bc28c3b988a14a P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:00.789901 17281 raft_consensus.cc:374] T 5a1b7f0005e54894b0bc28c3b988a14a P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:00.789997 17281 raft_consensus.cc:729] T 5a1b7f0005e54894b0bc28c3b988a14a P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 02:00:00.789964 17269 raft_consensus.cc:348] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:00.790194 17269 raft_consensus.cc:374] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:00.790230 17269 raft_consensus.cc:729] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 02:00:00.790400 17269 consensus_queue.cc:260] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:00.792311 17269 ts_tablet_manager.cc:1398] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:00.792863 17281 consensus_queue.cc:260] T 5a1b7f0005e54894b0bc28c3b988a14a P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:00.793278 17281 ts_tablet_manager.cc:1398] T 5a1b7f0005e54894b0bc28c3b988a14a P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:00:00.793895 17281 tablet_bootstrap.cc:492] T 90600b5313d64a10bf00fc1effd176e6 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 02:00:00.795066 17276 tablet_bootstrap.cc:492] T 5a1b7f0005e54894b0bc28c3b988a14a P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 02:00:00.795188 17276 ts_tablet_manager.cc:1367] T 5a1b7f0005e54894b0bc28c3b988a14a P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20240503 02:00:00.795805 17276 raft_consensus.cc:348] T 5a1b7f0005e54894b0bc28c3b988a14a P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:00.795923 17276 raft_consensus.cc:374] T 5a1b7f0005e54894b0bc28c3b988a14a P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:00.795980 17276 raft_consensus.cc:729] T 5a1b7f0005e54894b0bc28c3b988a14a P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 02:00:00.796123 17276 consensus_queue.cc:260] T 5a1b7f0005e54894b0bc28c3b988a14a P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:00.796511 17276 ts_tablet_manager.cc:1398] T 5a1b7f0005e54894b0bc28c3b988a14a P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:00.804041 17276 tablet_bootstrap.cc:492] T 90600b5313d64a10bf00fc1effd176e6 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 02:00:00.805940 17276 tablet_bootstrap.cc:654] T 90600b5313d64a10bf00fc1effd176e6 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:00.809047 17276 tablet_bootstrap.cc:492] T 90600b5313d64a10bf00fc1effd176e6 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 02:00:00.809252 17276 ts_tablet_manager.cc:1367] T 90600b5313d64a10bf00fc1effd176e6 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240503 02:00:00.809451 17281 tablet_bootstrap.cc:654] T 90600b5313d64a10bf00fc1effd176e6 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:00.809844 17276 raft_consensus.cc:348] T 90600b5313d64a10bf00fc1effd176e6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:00.809983 17276 raft_consensus.cc:374] T 90600b5313d64a10bf00fc1effd176e6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:00.810025 17276 raft_consensus.cc:729] T 90600b5313d64a10bf00fc1effd176e6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 02:00:00.810166 17276 consensus_queue.cc:260] T 90600b5313d64a10bf00fc1effd176e6 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:00.810470 17276 ts_tablet_manager.cc:1398] T 90600b5313d64a10bf00fc1effd176e6 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:00.811233 17281 tablet_bootstrap.cc:492] T 90600b5313d64a10bf00fc1effd176e6 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 02:00:00.811347 17281 ts_tablet_manager.cc:1367] T 90600b5313d64a10bf00fc1effd176e6 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 02:00:00.811847 17281 raft_consensus.cc:348] T 90600b5313d64a10bf00fc1effd176e6 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:00.811976 17281 raft_consensus.cc:374] T 90600b5313d64a10bf00fc1effd176e6 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:00.812009 17281 raft_consensus.cc:729] T 90600b5313d64a10bf00fc1effd176e6 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 02:00:00.812153 17281 consensus_queue.cc:260] T 90600b5313d64a10bf00fc1effd176e6 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:00.812819 17281 ts_tablet_manager.cc:1398] T 90600b5313d64a10bf00fc1effd176e6 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:00.820971 16962 raft_consensus.cc:482] T 90600b5313d64a10bf00fc1effd176e6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:00.821167 16962 raft_consensus.cc:504] T 90600b5313d64a10bf00fc1effd176e6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:00.821826 16962 leader_election.cc:288] T 90600b5313d64a10bf00fc1effd176e6 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 02:00:00.822124 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90600b5313d64a10bf00fc1effd176e6" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 02:00:00.822265 7143 raft_consensus.cc:2454] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 02:00:00.823431 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90600b5313d64a10bf00fc1effd176e6" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 02:00:00.823580 9493 raft_consensus.cc:2454] T 90600b5313d64a10bf00fc1effd176e6 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 02:00:00.823994 9559 leader_election.cc:302] T 90600b5313d64a10bf00fc1effd176e6 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 02:00:00.824198 16102 raft_consensus.cc:2781] T 90600b5313d64a10bf00fc1effd176e6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:00.824270 16102 raft_consensus.cc:482] T 90600b5313d64a10bf00fc1effd176e6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:00.824319 16102 raft_consensus.cc:3037] T 90600b5313d64a10bf00fc1effd176e6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.825496 16102 raft_consensus.cc:504] T 90600b5313d64a10bf00fc1effd176e6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:00.825881 16102 leader_election.cc:288] T 90600b5313d64a10bf00fc1effd176e6 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Requested vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 02:00:00.826308 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90600b5313d64a10bf00fc1effd176e6" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 02:00:00.826438 9493 raft_consensus.cc:3037] T 90600b5313d64a10bf00fc1effd176e6 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.827334 9493 raft_consensus.cc:2454] T 90600b5313d64a10bf00fc1effd176e6 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 02:00:00.828480 9559 leader_election.cc:302] T 90600b5313d64a10bf00fc1effd176e6 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 02:00:00.828650 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90600b5313d64a10bf00fc1effd176e6" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 02:00:00.828811 7143 raft_consensus.cc:3037] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.829838 7143 raft_consensus.cc:2454] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 02:00:00.829902 16102 raft_consensus.cc:2781] T 90600b5313d64a10bf00fc1effd176e6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:00.830021 16102 raft_consensus.cc:686] T 90600b5313d64a10bf00fc1effd176e6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 LEADER]: Becoming Leader. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Running, Role: LEADER
I20240503 02:00:00.830240 16102 consensus_queue.cc:237] T 90600b5313d64a10bf00fc1effd176e6 P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:00.832708 17301 ts_tablet_manager.cc:902] T 8ca4e33409b74f359d3242248652d8bb P db1a0000aaa346f79c0aa4b6c5105191: Initiating tablet copy from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 02:00:00.833298 17301 tablet_copy_client.cc:308] T 8ca4e33409b74f359d3242248652d8bb P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41979
I20240503 02:00:00.833535 6741 catalog_manager.cc:5484] T 90600b5313d64a10bf00fc1effd176e6 P b7062fbbc2b54fff9e99ec4ea5562c76 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 02:00:00.835168 7165 tablet_copy_service.cc:140] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Received BeginTabletCopySession request for tablet 8ca4e33409b74f359d3242248652d8bb from peer db1a0000aaa346f79c0aa4b6c5105191 ({username='slave'} at 127.0.99.136:40777)
I20240503 02:00:00.835279 7165 tablet_copy_service.cc:161] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Beginning new tablet copy session on tablet 8ca4e33409b74f359d3242248652d8bb from peer db1a0000aaa346f79c0aa4b6c5105191 at {username='slave'} at 127.0.99.136:40777: session id = db1a0000aaa346f79c0aa4b6c5105191-8ca4e33409b74f359d3242248652d8bb
I20240503 02:00:00.836139 7165 tablet_copy_source_session.cc:205] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:00.836099 7143 consensus_queue.cc:237] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:00.837374 9493 raft_consensus.cc:1261] T 76e8697ac8624e03a46bddae35f4c9fa P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:00.837800 16780 consensus_queue.cc:1035] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 02:00:00.837960 7079 consensus_peers.cc:485] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): Couldn't send request to peer db1a0000aaa346f79c0aa4b6c5105191. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 76e8697ac8624e03a46bddae35f4c9fa. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:00.838414 7274 raft_consensus.cc:1261] T 76e8697ac8624e03a46bddae35f4c9fa P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:00.838999 15159 consensus_queue.cc:1035] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:00.840694 17301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ca4e33409b74f359d3242248652d8bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:00.841650 15159 raft_consensus.cc:2932] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:00.842098 7274 raft_consensus.cc:2932] T 76e8697ac8624e03a46bddae35f4c9fa P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:00.842984 17301 tablet_copy_client.cc:783] T 8ca4e33409b74f359d3242248652d8bb P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:00.843220 17301 tablet_copy_client.cc:647] T 8ca4e33409b74f359d3242248652d8bb P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:00.844940 9493 raft_consensus.cc:2932] T 76e8697ac8624e03a46bddae35f4c9fa P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:00.846823 6741 catalog_manager.cc:5484] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index 5 to 6, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:00.852211 17301 tablet_copy_client.cc:523] T 8ca4e33409b74f359d3242248652d8bb P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:00.853905 17301 tablet_bootstrap.cc:492] T 8ca4e33409b74f359d3242248652d8bb P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 02:00:00.858691 17301 tablet_bootstrap.cc:492] T 8ca4e33409b74f359d3242248652d8bb P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:00.859210 17301 tablet_bootstrap.cc:492] T 8ca4e33409b74f359d3242248652d8bb P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap complete.
I20240503 02:00:00.859462 17301 ts_tablet_manager.cc:1367] T 8ca4e33409b74f359d3242248652d8bb P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20240503 02:00:00.859958 17301 raft_consensus.cc:348] T 8ca4e33409b74f359d3242248652d8bb P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:00.860064 17301 raft_consensus.cc:729] T 8ca4e33409b74f359d3242248652d8bb P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 02:00:00.860179 17301 consensus_queue.cc:260] T 8ca4e33409b74f359d3242248652d8bb P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:00.860641 17301 ts_tablet_manager.cc:1398] T 8ca4e33409b74f359d3242248652d8bb P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:00.861294 7165 tablet_copy_service.cc:342] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Request end of tablet copy session db1a0000aaa346f79c0aa4b6c5105191-8ca4e33409b74f359d3242248652d8bb received from {username='slave'} at 127.0.99.136:40777
I20240503 02:00:00.861385 7165 tablet_copy_service.cc:433] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: ending tablet copy session db1a0000aaa346f79c0aa4b6c5105191-8ca4e33409b74f359d3242248652d8bb on tablet 8ca4e33409b74f359d3242248652d8bb with peer db1a0000aaa346f79c0aa4b6c5105191
I20240503 02:00:00.874151 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:00.881182 17274 tablet.cc:1722] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:00.881558 17274 tablet.cc:1620] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 02:00:00.881593 17283 tablet.cc:1722] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:00.881899 17283 tablet.cc:1620] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 02:00:00.882912 17278 tablet.cc:1722] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:00.883404 17278 tablet.cc:1620] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: MemRowSet was empty: no flush needed.
I20240503 02:00:00.890374 6740 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=090454629ddd48d0b5d8047a6b6d6976] alter complete (version 2)
I20240503 02:00:00.891152 7145 consensus_queue.cc:237] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:00.892659 9625 raft_consensus.cc:1261] T d80398c9a264430197262bb42241d58d P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:00.893191 7012 raft_consensus.cc:1261] T d80398c9a264430197262bb42241d58d P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:00.893400 16780 consensus_queue.cc:1035] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:00.893642 17121 consensus_queue.cc:1035] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:00:00.895071 7078 consensus_peers.cc:485] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d80398c9a264430197262bb42241d58d. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:00.900488 16780 raft_consensus.cc:2932] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 02:00:00.901019 7012 raft_consensus.cc:2932] T d80398c9a264430197262bb42241d58d P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 02:00:00.903483 9625 raft_consensus.cc:2932] T d80398c9a264430197262bb42241d58d P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 02:00:00.905222 6734 catalog_manager.cc:5484] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index -1 to 2, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 02:00:00.930269 17144 raft_consensus.cc:482] T 9d44274085b9441c9f15676d842a0710 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:00.930387 17144 raft_consensus.cc:504] T 9d44274085b9441c9f15676d842a0710 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:00.930830 17144 leader_election.cc:288] T 9d44274085b9441c9f15676d842a0710 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 02:00:00.931171 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d44274085b9441c9f15676d842a0710" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 02:00:00.931255 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d44274085b9441c9f15676d842a0710" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 02:00:00.931314 7014 raft_consensus.cc:2454] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 02:00:00.931414 9755 raft_consensus.cc:2454] T 9d44274085b9441c9f15676d842a0710 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 02:00:00.931602 9296 leader_election.cc:302] T 9d44274085b9441c9f15676d842a0710 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 02:00:00.931766 17144 raft_consensus.cc:2781] T 9d44274085b9441c9f15676d842a0710 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:00.931818 17144 raft_consensus.cc:482] T 9d44274085b9441c9f15676d842a0710 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:00.931867 17144 raft_consensus.cc:3037] T 9d44274085b9441c9f15676d842a0710 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.933051 17144 raft_consensus.cc:504] T 9d44274085b9441c9f15676d842a0710 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:00.933432 17144 leader_election.cc:288] T 9d44274085b9441c9f15676d842a0710 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Requested vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 02:00:00.933691 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d44274085b9441c9f15676d842a0710" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 02:00:00.933802 9755 raft_consensus.cc:3037] T 9d44274085b9441c9f15676d842a0710 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.933869 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d44274085b9441c9f15676d842a0710" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 02:00:00.933975 7014 raft_consensus.cc:3037] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:00.934885 9755 raft_consensus.cc:2454] T 9d44274085b9441c9f15676d842a0710 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 02:00:00.934986 7014 raft_consensus.cc:2454] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 02:00:00.935357 9297 leader_election.cc:302] T 9d44274085b9441c9f15676d842a0710 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 02:00:00.935514 17144 raft_consensus.cc:2781] T 9d44274085b9441c9f15676d842a0710 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:00.935599 17144 raft_consensus.cc:686] T 9d44274085b9441c9f15676d842a0710 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Running, Role: LEADER
I20240503 02:00:00.935772 17144 consensus_queue.cc:237] T 9d44274085b9441c9f15676d842a0710 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:00.936864 9755 consensus_queue.cc:237] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:00.938728 7014 raft_consensus.cc:1261] T b3ce9de18bda410aa204bf7c1c32e1b7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:00.939129 9493 raft_consensus.cc:1261] T b3ce9de18bda410aa204bf7c1c32e1b7 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:00.939232 16785 consensus_queue.cc:1035] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:00.939390 6741 catalog_manager.cc:5484] T 9d44274085b9441c9f15676d842a0710 P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:00.939574 17266 consensus_queue.cc:1035] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:00:00.940702 9693 consensus_peers.cc:485] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): Couldn't send request to peer 5dcc33cc621a4cbbae155f3230fb7c46. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b3ce9de18bda410aa204bf7c1c32e1b7. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:00.942122 16785 raft_consensus.cc:2932] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:00.942705 9493 raft_consensus.cc:2932] T b3ce9de18bda410aa204bf7c1c32e1b7 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:00.944573 7014 raft_consensus.cc:2932] T b3ce9de18bda410aa204bf7c1c32e1b7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:00.945890 6741 catalog_manager.cc:5484] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:00.968849 7274 consensus_queue.cc:237] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:00.972610 7145 raft_consensus.cc:1261] T e811662836164c3fb29a1b12ace7eb41 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:00.972666 9493 raft_consensus.cc:1261] T e811662836164c3fb29a1b12ace7eb41 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 02:00:00.972771 7211 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e811662836164c3fb29a1b12ace7eb41. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:00.973095 16597 consensus_queue.cc:1035] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:00.974753 16597 consensus_queue.cc:1035] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:00.976996 16782 raft_consensus.cc:2932] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:00.977386 7145 raft_consensus.cc:2932] T e811662836164c3fb29a1b12ace7eb41 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:00.980549 6734 catalog_manager.cc:5484] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: config changed from index 5 to 6, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:00.981530 9493 raft_consensus.cc:2932] T e811662836164c3fb29a1b12ace7eb41 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:01.001188 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:01.008985 6881 consensus_queue.cc:237] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:01.009351 17274 tablet.cc:1722] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:01.009961 17278 tablet.cc:1722] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:01.011126 17383 ts_tablet_manager.cc:902] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef: Initiating tablet copy from peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:01.011132 9363 raft_consensus.cc:1261] T f8d3d130f4a44904924f7ccf4316fc75 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:01.011370 17283 tablet.cc:1722] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:01.012080 16401 consensus_queue.cc:1035] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:01.014463 9757 raft_consensus.cc:1261] T f8d3d130f4a44904924f7ccf4316fc75 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:01.014743 6817 consensus_peers.cc:485] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): Couldn't send request to peer 1d0e738c34af40fa9950c6f086619b3a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8d3d130f4a44904924f7ccf4316fc75. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:01.015105 16401 consensus_queue.cc:1035] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:01.019428 6734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=090454629ddd48d0b5d8047a6b6d6976] alter complete (version 3)
I20240503 02:00:01.020589 17383 tablet_copy_client.cc:308] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:37387
I20240503 02:00:01.021198 9383 tablet_copy_service.cc:140] P 12e8c683db054df3b75c6285b98a5ad8: Received BeginTabletCopySession request for tablet 6fd0378b2cff42a681361930f487d834 from peer 63331f455be049c1bdd6c23c59ac1bef ({username='slave'} at 127.0.99.130:52605)
I20240503 02:00:01.021310 9383 tablet_copy_service.cc:161] P 12e8c683db054df3b75c6285b98a5ad8: Beginning new tablet copy session on tablet 6fd0378b2cff42a681361930f487d834 from peer 63331f455be049c1bdd6c23c59ac1bef at {username='slave'} at 127.0.99.130:52605: session id = 63331f455be049c1bdd6c23c59ac1bef-6fd0378b2cff42a681361930f487d834
I20240503 02:00:01.021878 17203 raft_consensus.cc:2932] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:01.022257 9383 tablet_copy_source_session.cc:205] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:01.022594 9363 raft_consensus.cc:2932] T f8d3d130f4a44904924f7ccf4316fc75 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:01.023955 17383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd0378b2cff42a681361930f487d834. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.025501 9757 raft_consensus.cc:2932] T f8d3d130f4a44904924f7ccf4316fc75 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:01.026243 17383 tablet_copy_client.cc:783] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:01.026402 17383 tablet_copy_client.cc:647] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:01.027891 17383 tablet_copy_client.cc:523] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:01.029412 17383 tablet_bootstrap.cc:492] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 02:00:01.029577 6741 catalog_manager.cc:5484] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:01.036229 17383 tablet_bootstrap.cc:492] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:01.036523 17383 tablet_bootstrap.cc:492] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap complete.
I20240503 02:00:01.036710 17383 ts_tablet_manager.cc:1367] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 02:00:01.037286 17383 raft_consensus.cc:348] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:01.037436 17383 raft_consensus.cc:729] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.037622 17383 consensus_queue.cc:260] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:01.038230 17383 ts_tablet_manager.cc:1398] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:01.039235 9383 tablet_copy_service.cc:342] P 12e8c683db054df3b75c6285b98a5ad8: Request end of tablet copy session 63331f455be049c1bdd6c23c59ac1bef-6fd0378b2cff42a681361930f487d834 received from {username='slave'} at 127.0.99.130:52605
I20240503 02:00:01.039378 9383 tablet_copy_service.cc:433] P 12e8c683db054df3b75c6285b98a5ad8: ending tablet copy session 63331f455be049c1bdd6c23c59ac1bef-6fd0378b2cff42a681361930f487d834 on tablet 6fd0378b2cff42a681361930f487d834 with peer 63331f455be049c1bdd6c23c59ac1bef
I20240503 02:00:01.059967 9363 consensus_queue.cc:237] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:01.061489 9493 raft_consensus.cc:1261] T 22b6ec13c4e04d6cbade150f0c644e52 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:01.061714 9298 consensus_peers.cc:485] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 22b6ec13c4e04d6cbade150f0c644e52. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:01.062070 7145 raft_consensus.cc:1261] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:01.062361 15917 consensus_queue.cc:1035] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:01.062649 16988 consensus_queue.cc:1035] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:01.065137 15917 raft_consensus.cc:2932] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:01.065587 7145 raft_consensus.cc:2932] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:01.067076 9493 raft_consensus.cc:2932] T 22b6ec13c4e04d6cbade150f0c644e52 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:01.069391 6735 catalog_manager.cc:5484] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index -1 to 2, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:01.074338 17415 ts_tablet_manager.cc:902] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46: Initiating tablet copy from peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:01.075054 17415 tablet_copy_client.cc:308] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:37387
I20240503 02:00:01.075760 9383 tablet_copy_service.cc:140] P 12e8c683db054df3b75c6285b98a5ad8: Received BeginTabletCopySession request for tablet 9b8976486acf49d5a9d0d496379ee2a6 from peer 5dcc33cc621a4cbbae155f3230fb7c46 ({username='slave'} at 127.0.99.129:37661)
I20240503 02:00:01.075891 9383 tablet_copy_service.cc:161] P 12e8c683db054df3b75c6285b98a5ad8: Beginning new tablet copy session on tablet 9b8976486acf49d5a9d0d496379ee2a6 from peer 5dcc33cc621a4cbbae155f3230fb7c46 at {username='slave'} at 127.0.99.129:37661: session id = 5dcc33cc621a4cbbae155f3230fb7c46-9b8976486acf49d5a9d0d496379ee2a6
I20240503 02:00:01.076942 9383 tablet_copy_source_session.cc:205] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:01.078611 17415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b8976486acf49d5a9d0d496379ee2a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.082607 17415 tablet_copy_client.cc:783] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:01.082895 17415 tablet_copy_client.cc:647] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:01.084367 17415 tablet_copy_client.cc:523] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:01.086390 17415 tablet_bootstrap.cc:492] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 02:00:01.094306 17415 tablet_bootstrap.cc:492] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:01.094784 17415 tablet_bootstrap.cc:492] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap complete.
I20240503 02:00:01.094978 17415 ts_tablet_manager.cc:1367] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s
I20240503 02:00:01.095479 17415 raft_consensus.cc:348] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:01.095613 17415 raft_consensus.cc:729] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.095753 17415 consensus_queue.cc:260] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:01.096053 17415 ts_tablet_manager.cc:1398] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:01.096912 9383 tablet_copy_service.cc:342] P 12e8c683db054df3b75c6285b98a5ad8: Request end of tablet copy session 5dcc33cc621a4cbbae155f3230fb7c46-9b8976486acf49d5a9d0d496379ee2a6 received from {username='slave'} at 127.0.99.129:37661
I20240503 02:00:01.097003 9383 tablet_copy_service.cc:433] P 12e8c683db054df3b75c6285b98a5ad8: ending tablet copy session 5dcc33cc621a4cbbae155f3230fb7c46-9b8976486acf49d5a9d0d496379ee2a6 on tablet 9b8976486acf49d5a9d0d496379ee2a6 with peer 5dcc33cc621a4cbbae155f3230fb7c46
I20240503 02:00:01.111332 16102 raft_consensus.cc:482] T 5a1b7f0005e54894b0bc28c3b988a14a P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:01.111524 16102 raft_consensus.cc:504] T 5a1b7f0005e54894b0bc28c3b988a14a P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:01.112303 16102 leader_election.cc:288] T 5a1b7f0005e54894b0bc28c3b988a14a P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 02:00:01.113581 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1b7f0005e54894b0bc28c3b988a14a" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 02:00:01.113762 9493 raft_consensus.cc:2454] T 5a1b7f0005e54894b0bc28c3b988a14a P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 02:00:01.114586 9559 leader_election.cc:302] T 5a1b7f0005e54894b0bc28c3b988a14a P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 02:00:01.114675 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1b7f0005e54894b0bc28c3b988a14a" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 02:00:01.114810 7145 raft_consensus.cc:2454] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 02:00:01.116078 16102 raft_consensus.cc:2781] T 5a1b7f0005e54894b0bc28c3b988a14a P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:01.116148 16102 raft_consensus.cc:482] T 5a1b7f0005e54894b0bc28c3b988a14a P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:01.116186 16102 raft_consensus.cc:3037] T 5a1b7f0005e54894b0bc28c3b988a14a P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:01.117321 16102 raft_consensus.cc:504] T 5a1b7f0005e54894b0bc28c3b988a14a P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:01.117758 16102 leader_election.cc:288] T 5a1b7f0005e54894b0bc28c3b988a14a P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Requested vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 02:00:01.118376 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1b7f0005e54894b0bc28c3b988a14a" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 02:00:01.118507 7145 raft_consensus.cc:3037] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:01.118534 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1b7f0005e54894b0bc28c3b988a14a" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 02:00:01.118652 9493 raft_consensus.cc:3037] T 5a1b7f0005e54894b0bc28c3b988a14a P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:01.119648 9493 raft_consensus.cc:2454] T 5a1b7f0005e54894b0bc28c3b988a14a P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 02:00:01.119648 7145 raft_consensus.cc:2454] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 02:00:01.120332 9561 leader_election.cc:302] T 5a1b7f0005e54894b0bc28c3b988a14a P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 02:00:01.121649 16102 raft_consensus.cc:2781] T 5a1b7f0005e54894b0bc28c3b988a14a P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:01.122041 16102 raft_consensus.cc:686] T 5a1b7f0005e54894b0bc28c3b988a14a P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 LEADER]: Becoming Leader. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Running, Role: LEADER
I20240503 02:00:01.122215 16102 consensus_queue.cc:237] T 5a1b7f0005e54894b0bc28c3b988a14a P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:01.125041 6741 catalog_manager.cc:5484] T 5a1b7f0005e54894b0bc28c3b988a14a P b7062fbbc2b54fff9e99ec4ea5562c76 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:01.129315 6734 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:01.129451 6734 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:01.131328 17301 ts_tablet_manager.cc:902] T 4bd8226227c348d5a5d4d8ff28959d27 P db1a0000aaa346f79c0aa4b6c5105191: Initiating tablet copy from peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 02:00:01.131836 17301 tablet_copy_client.cc:308] T 4bd8226227c348d5a5d4d8ff28959d27 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:42943
I20240503 02:00:01.132989 6903 tablet_copy_service.cc:140] P 5dcc33cc621a4cbbae155f3230fb7c46: Received BeginTabletCopySession request for tablet 4bd8226227c348d5a5d4d8ff28959d27 from peer db1a0000aaa346f79c0aa4b6c5105191 ({username='slave'} at 127.0.99.136:40711)
I20240503 02:00:01.133090 6903 tablet_copy_service.cc:161] P 5dcc33cc621a4cbbae155f3230fb7c46: Beginning new tablet copy session on tablet 4bd8226227c348d5a5d4d8ff28959d27 from peer db1a0000aaa346f79c0aa4b6c5105191 at {username='slave'} at 127.0.99.136:40711: session id = db1a0000aaa346f79c0aa4b6c5105191-4bd8226227c348d5a5d4d8ff28959d27
I20240503 02:00:01.134126 6903 tablet_copy_source_session.cc:205] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:01.135924 6734 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet fd8202c2dfbe4a24b8b6324ca3e31cf1
I20240503 02:00:01.136777 9342 tablet_service.cc:1514] Processing DeleteTablet for tablet fd8202c2dfbe4a24b8b6324ca3e31cf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:01 UTC) from {username='slave'} at 127.0.0.1:36040
I20240503 02:00:01.136929 7124 tablet_service.cc:1514] Processing DeleteTablet for tablet fd8202c2dfbe4a24b8b6324ca3e31cf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:01 UTC) from {username='slave'} at 127.0.0.1:50370
I20240503 02:00:01.137094 9736 tablet_service.cc:1514] Processing DeleteTablet for tablet fd8202c2dfbe4a24b8b6324ca3e31cf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:01 UTC) from {username='slave'} at 127.0.0.1:48152
I20240503 02:00:01.138736 17435 tablet_replica.cc:330] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8: stopping tablet replica
I20240503 02:00:01.138942 17435 raft_consensus.cc:2229] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:01.139042 17301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bd8226227c348d5a5d4d8ff28959d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.139261 17435 raft_consensus.cc:2258] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.141325 17301 tablet_copy_client.cc:783] T 4bd8226227c348d5a5d4d8ff28959d27 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:01.141494 17301 tablet_copy_client.cc:647] T 4bd8226227c348d5a5d4d8ff28959d27 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:01.142057 17435 ts_tablet_manager.cc:1875] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:01.144055 17301 tablet_copy_client.cc:523] T 4bd8226227c348d5a5d4d8ff28959d27 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:01.144850 17435 ts_tablet_manager.cc:1888] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:01.145009 17435 log.cc:1196] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/fd8202c2dfbe4a24b8b6324ca3e31cf1
I20240503 02:00:01.145381 17435 ts_tablet_manager.cc:1909] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P 12e8c683db054df3b75c6285b98a5ad8: Deleting consensus metadata
I20240503 02:00:01.145821 17301 tablet_bootstrap.cc:492] T 4bd8226227c348d5a5d4d8ff28959d27 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 02:00:01.146775 6727 catalog_manager.cc:4830] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): tablet fd8202c2dfbe4a24b8b6324ca3e31cf1 (table rebalancer_dynamic_table_new_name [id=090454629ddd48d0b5d8047a6b6d6976]) successfully deleted
I20240503 02:00:01.157467 17383 ts_tablet_manager.cc:902] T b93fe374d2a9496a847480b7824a5937 P 63331f455be049c1bdd6c23c59ac1bef: Initiating tablet copy from peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:01.158026 17383 tablet_copy_client.cc:308] T b93fe374d2a9496a847480b7824a5937 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:46087
I20240503 02:00:01.150758 17436 tablet_replica.cc:330] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 02:00:01.158500 9646 tablet_copy_service.cc:140] P b7062fbbc2b54fff9e99ec4ea5562c76: Received BeginTabletCopySession request for tablet b93fe374d2a9496a847480b7824a5937 from peer 63331f455be049c1bdd6c23c59ac1bef ({username='slave'} at 127.0.99.130:43977)
I20240503 02:00:01.158639 9646 tablet_copy_service.cc:161] P b7062fbbc2b54fff9e99ec4ea5562c76: Beginning new tablet copy session on tablet b93fe374d2a9496a847480b7824a5937 from peer 63331f455be049c1bdd6c23c59ac1bef at {username='slave'} at 127.0.99.130:43977: session id = 63331f455be049c1bdd6c23c59ac1bef-b93fe374d2a9496a847480b7824a5937
I20240503 02:00:01.153445 17439 ts_tablet_manager.cc:902] T 9e21d082e40540d4bb908fc9e90055c9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Initiating tablet copy from peer db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 02:00:01.158839 17436 raft_consensus.cc:2229] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.159024 17436 raft_consensus.cc:2258] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.159261 17439 tablet_copy_client.cc:308] T 9e21d082e40540d4bb908fc9e90055c9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:42053
I20240503 02:00:01.159868 17436 ts_tablet_manager.cc:1875] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:01.159883 9646 tablet_copy_source_session.cc:205] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:01.159981 9777 tablet_copy_service.cc:140] P db1a0000aaa346f79c0aa4b6c5105191: Received BeginTabletCopySession request for tablet 9e21d082e40540d4bb908fc9e90055c9 from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a ({username='slave'} at 127.0.99.131:54483)
I20240503 02:00:01.160094 9777 tablet_copy_service.cc:161] P db1a0000aaa346f79c0aa4b6c5105191: Beginning new tablet copy session on tablet 9e21d082e40540d4bb908fc9e90055c9 from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a at {username='slave'} at 127.0.99.131:54483: session id = c688f3df56ac4b6d9b0c30fa4cc8ca2a-9e21d082e40540d4bb908fc9e90055c9
I20240503 02:00:01.161087 9777 tablet_copy_source_session.cc:205] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:01.161965 17439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e21d082e40540d4bb908fc9e90055c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.164263 17439 tablet_copy_client.cc:783] T 9e21d082e40540d4bb908fc9e90055c9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:01.164299 17437 tablet_replica.cc:330] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P db1a0000aaa346f79c0aa4b6c5105191: stopping tablet replica
I20240503 02:00:01.164433 17439 tablet_copy_client.cc:647] T 9e21d082e40540d4bb908fc9e90055c9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:01.164444 17437 raft_consensus.cc:2229] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.164566 17437 raft_consensus.cc:2258] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.165196 17383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b93fe374d2a9496a847480b7824a5937. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.165704 17437 ts_tablet_manager.cc:1875] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P db1a0000aaa346f79c0aa4b6c5105191: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:01.167127 17383 tablet_copy_client.cc:783] T b93fe374d2a9496a847480b7824a5937 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:01.167292 17383 tablet_copy_client.cc:647] T b93fe374d2a9496a847480b7824a5937 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:01.167850 17437 ts_tablet_manager.cc:1888] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P db1a0000aaa346f79c0aa4b6c5105191: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:01.168709 17383 tablet_copy_client.cc:523] T b93fe374d2a9496a847480b7824a5937 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:01.168836 17439 tablet_copy_client.cc:523] T 9e21d082e40540d4bb908fc9e90055c9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:01.170182 17383 tablet_bootstrap.cc:492] T b93fe374d2a9496a847480b7824a5937 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 02:00:01.170182 17439 tablet_bootstrap.cc:492] T 9e21d082e40540d4bb908fc9e90055c9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 02:00:01.170320 17436 ts_tablet_manager.cc:1888] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:01.172959 17436 log.cc:1196] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/fd8202c2dfbe4a24b8b6324ca3e31cf1
I20240503 02:00:01.173743 17436 ts_tablet_manager.cc:1909] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting consensus metadata
I20240503 02:00:01.175122 17437 log.cc:1196] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P db1a0000aaa346f79c0aa4b6c5105191: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/fd8202c2dfbe4a24b8b6324ca3e31cf1
I20240503 02:00:01.175253 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet fd8202c2dfbe4a24b8b6324ca3e31cf1 (table rebalancer_dynamic_table_new_name [id=090454629ddd48d0b5d8047a6b6d6976]) successfully deleted
I20240503 02:00:01.175493 17439 tablet_bootstrap.cc:492] T 9e21d082e40540d4bb908fc9e90055c9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:01.175568 17437 ts_tablet_manager.cc:1909] T fd8202c2dfbe4a24b8b6324ca3e31cf1 P db1a0000aaa346f79c0aa4b6c5105191: Deleting consensus metadata
I20240503 02:00:01.174844 9757 consensus_queue.cc:237] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:01.175750 17439 tablet_bootstrap.cc:492] T 9e21d082e40540d4bb908fc9e90055c9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap complete.
I20240503 02:00:01.175915 17439 ts_tablet_manager.cc:1367] T 9e21d082e40540d4bb908fc9e90055c9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:00:01.176379 17439 raft_consensus.cc:348] T 9e21d082e40540d4bb908fc9e90055c9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:01.176512 17439 raft_consensus.cc:729] T 9e21d082e40540d4bb908fc9e90055c9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.176656 17439 consensus_queue.cc:260] T 9e21d082e40540d4bb908fc9e90055c9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:01.176967 17439 ts_tablet_manager.cc:1398] T 9e21d082e40540d4bb908fc9e90055c9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:01.177070 6726 catalog_manager.cc:4830] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): tablet fd8202c2dfbe4a24b8b6324ca3e31cf1 (table rebalancer_dynamic_table_new_name [id=090454629ddd48d0b5d8047a6b6d6976]) successfully deleted
I20240503 02:00:01.178339 17415 ts_tablet_manager.cc:902] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46: Initiating tablet copy from peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 02:00:01.178716 7014 raft_consensus.cc:1261] T e8cb37039be049cf9c4e53bbcfbd80d9 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:01.179091 17383 tablet_bootstrap.cc:492] T b93fe374d2a9496a847480b7824a5937 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:01.179453 17415 tablet_copy_client.cc:308] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:43471
I20240503 02:00:01.179479 17383 tablet_bootstrap.cc:492] T b93fe374d2a9496a847480b7824a5937 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap complete.
I20240503 02:00:01.179669 17383 ts_tablet_manager.cc:1367] T b93fe374d2a9496a847480b7824a5937 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20240503 02:00:01.179904 9514 tablet_copy_service.cc:140] P bb5364d730cb4489a0bfc6c270b7450a: Received BeginTabletCopySession request for tablet 5089809a9f3846f5b30397b1062d4e55 from peer 5dcc33cc621a4cbbae155f3230fb7c46 ({username='slave'} at 127.0.99.129:38243)
I20240503 02:00:01.180009 9514 tablet_copy_service.cc:161] P bb5364d730cb4489a0bfc6c270b7450a: Beginning new tablet copy session on tablet 5089809a9f3846f5b30397b1062d4e55 from peer 5dcc33cc621a4cbbae155f3230fb7c46 at {username='slave'} at 127.0.99.129:38243: session id = 5dcc33cc621a4cbbae155f3230fb7c46-5089809a9f3846f5b30397b1062d4e55
I20240503 02:00:01.179980 17266 consensus_queue.cc:1035] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:01.180127 17383 raft_consensus.cc:348] T b93fe374d2a9496a847480b7824a5937 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:01.180249 17383 raft_consensus.cc:729] T b93fe374d2a9496a847480b7824a5937 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.180387 17383 consensus_queue.cc:260] T b93fe374d2a9496a847480b7824a5937 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:01.180663 17383 ts_tablet_manager.cc:1398] T b93fe374d2a9496a847480b7824a5937 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:01.181072 9514 tablet_copy_source_session.cc:205] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:01.181270 9646 tablet_copy_service.cc:342] P b7062fbbc2b54fff9e99ec4ea5562c76: Request end of tablet copy session 63331f455be049c1bdd6c23c59ac1bef-b93fe374d2a9496a847480b7824a5937 received from {username='slave'} at 127.0.99.130:43977
I20240503 02:00:01.181349 9646 tablet_copy_service.cc:433] P b7062fbbc2b54fff9e99ec4ea5562c76: ending tablet copy session 63331f455be049c1bdd6c23c59ac1bef-b93fe374d2a9496a847480b7824a5937 on tablet b93fe374d2a9496a847480b7824a5937 with peer 63331f455be049c1bdd6c23c59ac1bef
I20240503 02:00:01.181399 9777 tablet_copy_service.cc:342] P db1a0000aaa346f79c0aa4b6c5105191: Request end of tablet copy session c688f3df56ac4b6d9b0c30fa4cc8ca2a-9e21d082e40540d4bb908fc9e90055c9 received from {username='slave'} at 127.0.99.131:54483
I20240503 02:00:01.181463 9777 tablet_copy_service.cc:433] P db1a0000aaa346f79c0aa4b6c5105191: ending tablet copy session c688f3df56ac4b6d9b0c30fa4cc8ca2a-9e21d082e40540d4bb908fc9e90055c9 on tablet 9e21d082e40540d4bb908fc9e90055c9 with peer c688f3df56ac4b6d9b0c30fa4cc8ca2a
I20240503 02:00:01.182360 17439 ts_tablet_manager.cc:902] T 0bbbd17c78bc4e329a3a7c7008a34d86 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Initiating tablet copy from peer db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 02:00:01.182811 17439 tablet_copy_client.cc:308] T 0bbbd17c78bc4e329a3a7c7008a34d86 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:42053
I20240503 02:00:01.183367 9777 tablet_copy_service.cc:140] P db1a0000aaa346f79c0aa4b6c5105191: Received BeginTabletCopySession request for tablet 0bbbd17c78bc4e329a3a7c7008a34d86 from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a ({username='slave'} at 127.0.99.131:54483)
I20240503 02:00:01.183454 9777 tablet_copy_service.cc:161] P db1a0000aaa346f79c0aa4b6c5105191: Beginning new tablet copy session on tablet 0bbbd17c78bc4e329a3a7c7008a34d86 from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a at {username='slave'} at 127.0.99.131:54483: session id = c688f3df56ac4b6d9b0c30fa4cc8ca2a-0bbbd17c78bc4e329a3a7c7008a34d86
I20240503 02:00:01.183908 9624 raft_consensus.cc:1261] T e8cb37039be049cf9c4e53bbcfbd80d9 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:01.184329 9777 tablet_copy_source_session.cc:205] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191: Tablet Copy: opened 0 blocks and 1 log segments
W20240503 02:00:01.184407 9692 consensus_peers.cc:485] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): Couldn't send request to peer c688f3df56ac4b6d9b0c30fa4cc8ca2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e8cb37039be049cf9c4e53bbcfbd80d9. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:01.184592 17183 consensus_queue.cc:1035] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:01.185560 17439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bbbd17c78bc4e329a3a7c7008a34d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.187237 17415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5089809a9f3846f5b30397b1062d4e55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.187914 17439 tablet_copy_client.cc:783] T 0bbbd17c78bc4e329a3a7c7008a34d86 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:01.188089 17439 tablet_copy_client.cc:647] T 0bbbd17c78bc4e329a3a7c7008a34d86 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:01.189924 17266 raft_consensus.cc:2932] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:01.190323 17439 tablet_copy_client.cc:523] T 0bbbd17c78bc4e329a3a7c7008a34d86 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:01.190424 9624 raft_consensus.cc:2932] T e8cb37039be049cf9c4e53bbcfbd80d9 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:01.191772 17439 tablet_bootstrap.cc:492] T 0bbbd17c78bc4e329a3a7c7008a34d86 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 02:00:01.194401 17415 tablet_copy_client.cc:783] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:01.195026 7014 raft_consensus.cc:2932] T e8cb37039be049cf9c4e53bbcfbd80d9 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:01.195483 6741 catalog_manager.cc:5484] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:01.199172 17415 tablet_copy_client.cc:647] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:01.201592 17415 tablet_copy_client.cc:523] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:01.203454 17301 tablet_bootstrap.cc:492] T 4bd8226227c348d5a5d4d8ff28959d27 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:01.203758 17301 tablet_bootstrap.cc:492] T 4bd8226227c348d5a5d4d8ff28959d27 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap complete.
I20240503 02:00:01.203876 17415 tablet_bootstrap.cc:492] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 02:00:01.203940 17301 ts_tablet_manager.cc:1367] T 4bd8226227c348d5a5d4d8ff28959d27 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.058s user 0.027s sys 0.000s
I20240503 02:00:01.204716 17301 raft_consensus.cc:348] T 4bd8226227c348d5a5d4d8ff28959d27 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:01.204859 17301 raft_consensus.cc:729] T 4bd8226227c348d5a5d4d8ff28959d27 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.205013 17301 consensus_queue.cc:260] T 4bd8226227c348d5a5d4d8ff28959d27 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:01.206275 17301 ts_tablet_manager.cc:1398] T 4bd8226227c348d5a5d4d8ff28959d27 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:00:01.206794 6903 tablet_copy_service.cc:342] P 5dcc33cc621a4cbbae155f3230fb7c46: Request end of tablet copy session db1a0000aaa346f79c0aa4b6c5105191-4bd8226227c348d5a5d4d8ff28959d27 received from {username='slave'} at 127.0.99.136:40711
I20240503 02:00:01.206871 6903 tablet_copy_service.cc:433] P 5dcc33cc621a4cbbae155f3230fb7c46: ending tablet copy session db1a0000aaa346f79c0aa4b6c5105191-4bd8226227c348d5a5d4d8ff28959d27 on tablet 4bd8226227c348d5a5d4d8ff28959d27 with peer db1a0000aaa346f79c0aa4b6c5105191
I20240503 02:00:01.208295 17439 tablet_bootstrap.cc:492] T 0bbbd17c78bc4e329a3a7c7008a34d86 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:01.208348 17415 tablet_bootstrap.cc:492] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:01.208535 17439 tablet_bootstrap.cc:492] T 0bbbd17c78bc4e329a3a7c7008a34d86 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap complete.
I20240503 02:00:01.208560 17415 tablet_bootstrap.cc:492] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap complete.
I20240503 02:00:01.208710 17415 ts_tablet_manager.cc:1367] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:00:01.208735 17439 ts_tablet_manager.cc:1367] T 0bbbd17c78bc4e329a3a7c7008a34d86 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.001s
I20240503 02:00:01.209152 17415 raft_consensus.cc:348] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:01.209154 17439 raft_consensus.cc:348] T 0bbbd17c78bc4e329a3a7c7008a34d86 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:01.209254 17415 raft_consensus.cc:729] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.209257 17439 raft_consensus.cc:729] T 0bbbd17c78bc4e329a3a7c7008a34d86 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.209360 17415 consensus_queue.cc:260] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:01.209427 17439 consensus_queue.cc:260] T 0bbbd17c78bc4e329a3a7c7008a34d86 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:01.209641 17415 ts_tablet_manager.cc:1398] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:01.209769 17439 ts_tablet_manager.cc:1398] T 0bbbd17c78bc4e329a3a7c7008a34d86 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:01.211068 9514 tablet_copy_service.cc:342] P bb5364d730cb4489a0bfc6c270b7450a: Request end of tablet copy session 5dcc33cc621a4cbbae155f3230fb7c46-5089809a9f3846f5b30397b1062d4e55 received from {username='slave'} at 127.0.99.129:38243
I20240503 02:00:01.211163 9514 tablet_copy_service.cc:433] P bb5364d730cb4489a0bfc6c270b7450a: ending tablet copy session 5dcc33cc621a4cbbae155f3230fb7c46-5089809a9f3846f5b30397b1062d4e55 on tablet 5089809a9f3846f5b30397b1062d4e55 with peer 5dcc33cc621a4cbbae155f3230fb7c46
I20240503 02:00:01.212003 9777 tablet_copy_service.cc:342] P db1a0000aaa346f79c0aa4b6c5105191: Request end of tablet copy session c688f3df56ac4b6d9b0c30fa4cc8ca2a-0bbbd17c78bc4e329a3a7c7008a34d86 received from {username='slave'} at 127.0.99.131:54483
I20240503 02:00:01.212123 9777 tablet_copy_service.cc:433] P db1a0000aaa346f79c0aa4b6c5105191: ending tablet copy session c688f3df56ac4b6d9b0c30fa4cc8ca2a-0bbbd17c78bc4e329a3a7c7008a34d86 on tablet 0bbbd17c78bc4e329a3a7c7008a34d86 with peer c688f3df56ac4b6d9b0c30fa4cc8ca2a
I20240503 02:00:01.229725 6882 consensus_queue.cc:237] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:01.231415 9624 raft_consensus.cc:1261] T fd29f3b059874b019153b64ef8aaaa48 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:01.231527 7014 raft_consensus.cc:1261] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:01.231807 17202 consensus_queue.cc:1035] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:01.231981 16673 consensus_queue.cc:1035] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:00:01.232267 6818 consensus_peers.cc:485] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd29f3b059874b019153b64ef8aaaa48. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:01.233506 16962 consensus_queue.cc:1035] T 90600b5313d64a10bf00fc1effd176e6 P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:01.234728 16673 raft_consensus.cc:2932] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:01.237694 7014 raft_consensus.cc:2932] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:01.238941 6741 catalog_manager.cc:5484] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index -1 to 2, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:01.239904 9624 raft_consensus.cc:2932] T fd29f3b059874b019153b64ef8aaaa48 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:01.241869 6734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:01.242094 16102 consensus_queue.cc:1035] T 90600b5313d64a10bf00fc1effd176e6 P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:01.250291 9472 tablet_service.cc:1467] Processing CreateTablet for tablet 4da3d94fdd1c42b2bc9cb08216bb84dc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3bbac19b98664efeaf8f0aafd4e7655f]), partition=
I20240503 02:00:01.250337 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 4da3d94fdd1c42b2bc9cb08216bb84dc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3bbac19b98664efeaf8f0aafd4e7655f]), partition=
I20240503 02:00:01.250577 9472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da3d94fdd1c42b2bc9cb08216bb84dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.250651 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da3d94fdd1c42b2bc9cb08216bb84dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.252476 17281 tablet_bootstrap.cc:492] T 4da3d94fdd1c42b2bc9cb08216bb84dc P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 02:00:01.253234 17122 tablet_bootstrap.cc:492] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 02:00:01.253475 17281 tablet_bootstrap.cc:654] T 4da3d94fdd1c42b2bc9cb08216bb84dc P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:01.253532 6863 tablet_service.cc:1467] Processing CreateTablet for tablet 4da3d94fdd1c42b2bc9cb08216bb84dc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3bbac19b98664efeaf8f0aafd4e7655f]), partition=
I20240503 02:00:01.253840 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da3d94fdd1c42b2bc9cb08216bb84dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.254206 17122 tablet_bootstrap.cc:654] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:01.255321 17122 tablet_bootstrap.cc:492] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 02:00:01.255410 17122 ts_tablet_manager.cc:1367] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:00:01.255894 17122 raft_consensus.cc:348] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:01.256011 17122 raft_consensus.cc:374] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:01.256048 17122 raft_consensus.cc:729] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.256155 17122 consensus_queue.cc:260] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:01.256202 17281 tablet_bootstrap.cc:492] T 4da3d94fdd1c42b2bc9cb08216bb84dc P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 02:00:01.256289 17281 ts_tablet_manager.cc:1367] T 4da3d94fdd1c42b2bc9cb08216bb84dc P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:01.256434 17122 ts_tablet_manager.cc:1398] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:01.256717 17281 raft_consensus.cc:348] T 4da3d94fdd1c42b2bc9cb08216bb84dc P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:01.256817 17281 raft_consensus.cc:374] T 4da3d94fdd1c42b2bc9cb08216bb84dc P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:01.256850 17281 raft_consensus.cc:729] T 4da3d94fdd1c42b2bc9cb08216bb84dc P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.256975 17281 consensus_queue.cc:260] T 4da3d94fdd1c42b2bc9cb08216bb84dc P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:01.257246 17281 ts_tablet_manager.cc:1398] T 4da3d94fdd1c42b2bc9cb08216bb84dc P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:01.257867 17495 tablet_bootstrap.cc:492] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 02:00:01.259979 17495 tablet_bootstrap.cc:654] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:01.262033 17495 tablet_bootstrap.cc:492] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 02:00:01.262197 17495 ts_tablet_manager.cc:1367] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:00:01.262784 17495 raft_consensus.cc:348] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:01.262995 17495 raft_consensus.cc:374] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:01.263092 17495 raft_consensus.cc:729] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.263271 17495 consensus_queue.cc:260] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:01.263717 17495 ts_tablet_manager.cc:1398] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:01.283622 7014 consensus_queue.cc:237] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:01.285781 9363 raft_consensus.cc:1261] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:01.286378 9757 raft_consensus.cc:1261] T aaeebd4c884445ba9f44ac9bfcb9c7ba P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:01.286583 16636 consensus_queue.cc:1035] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:00:01.286967 6948 consensus_peers.cc:485] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef -> Peer 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): Couldn't send request to peer 1d0e738c34af40fa9950c6f086619b3a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aaeebd4c884445ba9f44ac9bfcb9c7ba. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:01.287003 16636 consensus_queue.cc:1035] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:01.290627 16636 raft_consensus.cc:2932] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:01.291174 9757 raft_consensus.cc:2932] T aaeebd4c884445ba9f44ac9bfcb9c7ba P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:01.294301 6741 catalog_manager.cc:5484] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:01.295112 9363 raft_consensus.cc:2932] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:01.335600 9493 consensus_queue.cc:237] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:01.337249 6882 raft_consensus.cc:1261] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:00:01.337400 9427 consensus_peers.cc:485] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a -> Peer 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): Couldn't send request to peer 63331f455be049c1bdd6c23c59ac1bef. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e824fcc9aac449d9373655b304010e2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:01.337708 17529 consensus_queue.cc:1035] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:01.339270 9363 raft_consensus.cc:1261] T 3e824fcc9aac449d9373655b304010e2 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:00:01.339637 17529 consensus_queue.cc:1035] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:01.359184 17529 raft_consensus.cc:2932] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:01.361537 6882 raft_consensus.cc:2932] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:01.361984 16988 raft_consensus.cc:482] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:01.362071 16988 raft_consensus.cc:504] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:01.362417 16988 leader_election.cc:288] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 02:00:01.362784 6881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da3d94fdd1c42b2bc9cb08216bb84dc" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 02:00:01.362884 6881 raft_consensus.cc:2454] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 02:00:01.363111 6735 catalog_manager.cc:5484] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: config changed from index 3 to 4, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:01.363368 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da3d94fdd1c42b2bc9cb08216bb84dc" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 02:00:01.363484 9493 raft_consensus.cc:2454] T 4da3d94fdd1c42b2bc9cb08216bb84dc P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 02:00:01.363699 9299 leader_election.cc:302] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 02:00:01.364097 16988 raft_consensus.cc:2781] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:01.364166 16988 raft_consensus.cc:482] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:01.364200 16988 raft_consensus.cc:3037] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:01.365129 16988 raft_consensus.cc:504] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:01.365532 16988 leader_election.cc:288] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Requested vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 02:00:01.366264 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da3d94fdd1c42b2bc9cb08216bb84dc" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 02:00:01.366376 9493 raft_consensus.cc:3037] T 4da3d94fdd1c42b2bc9cb08216bb84dc P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:01.367483 9493 raft_consensus.cc:2454] T 4da3d94fdd1c42b2bc9cb08216bb84dc P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 02:00:01.368139 9296 leader_election.cc:302] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a; no voters:
I20240503 02:00:01.368302 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da3d94fdd1c42b2bc9cb08216bb84dc" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 02:00:01.368607 6882 raft_consensus.cc:3037] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:01.368821 16988 raft_consensus.cc:2781] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:01.368902 16988 raft_consensus.cc:686] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Running, Role: LEADER
I20240503 02:00:01.369227 16988 consensus_queue.cc:237] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:01.369761 6882 raft_consensus.cc:2454] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 02:00:01.373299 6734 catalog_manager.cc:5484] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:01.377125 9363 raft_consensus.cc:2932] T 3e824fcc9aac449d9373655b304010e2 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:01.396155 17301 ts_tablet_manager.cc:902] T 76e8697ac8624e03a46bddae35f4c9fa P db1a0000aaa346f79c0aa4b6c5105191: Initiating tablet copy from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 02:00:01.396692 17301 tablet_copy_client.cc:308] T 76e8697ac8624e03a46bddae35f4c9fa P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41979
I20240503 02:00:01.397162 7165 tablet_copy_service.cc:140] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Received BeginTabletCopySession request for tablet 76e8697ac8624e03a46bddae35f4c9fa from peer db1a0000aaa346f79c0aa4b6c5105191 ({username='slave'} at 127.0.99.136:40777)
I20240503 02:00:01.397280 7165 tablet_copy_service.cc:161] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Beginning new tablet copy session on tablet 76e8697ac8624e03a46bddae35f4c9fa from peer db1a0000aaa346f79c0aa4b6c5105191 at {username='slave'} at 127.0.99.136:40777: session id = db1a0000aaa346f79c0aa4b6c5105191-76e8697ac8624e03a46bddae35f4c9fa
I20240503 02:00:01.398355 7165 tablet_copy_source_session.cc:205] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:01.399871 17301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76e8697ac8624e03a46bddae35f4c9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.402174 17301 tablet_copy_client.cc:783] T 76e8697ac8624e03a46bddae35f4c9fa P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:01.402354 17301 tablet_copy_client.cc:647] T 76e8697ac8624e03a46bddae35f4c9fa P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:01.404796 7275 consensus_queue.cc:237] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:01.405287 17301 tablet_copy_client.cc:523] T 76e8697ac8624e03a46bddae35f4c9fa P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:01.406956 9624 raft_consensus.cc:1261] T 7046186cd7f540e7b0292da5f422a4a1 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:01.406941 7211 consensus_peers.cc:485] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): Couldn't send request to peer c688f3df56ac4b6d9b0c30fa4cc8ca2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7046186cd7f540e7b0292da5f422a4a1. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:01.406949 9493 raft_consensus.cc:1261] T 7046186cd7f540e7b0292da5f422a4a1 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:01.407413 17301 tablet_bootstrap.cc:492] T 76e8697ac8624e03a46bddae35f4c9fa P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 02:00:01.407475 17364 consensus_queue.cc:1035] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:01.407940 17364 consensus_queue.cc:1035] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:01.409894 16782 raft_consensus.cc:2932] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:01.410331 9624 raft_consensus.cc:2932] T 7046186cd7f540e7b0292da5f422a4a1 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:01.412448 9493 raft_consensus.cc:2932] T 7046186cd7f540e7b0292da5f422a4a1 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:01.414084 6735 catalog_manager.cc:5484] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:01.418795 17558 ts_tablet_manager.cc:902] T f8d3d130f4a44904924f7ccf4316fc75 P 1d0e738c34af40fa9950c6f086619b3a: Initiating tablet copy from peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 02:00:01.419435 17558 tablet_copy_client.cc:308] T f8d3d130f4a44904924f7ccf4316fc75 P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:42943
I20240503 02:00:01.420025 6903 tablet_copy_service.cc:140] P 5dcc33cc621a4cbbae155f3230fb7c46: Received BeginTabletCopySession request for tablet f8d3d130f4a44904924f7ccf4316fc75 from peer 1d0e738c34af40fa9950c6f086619b3a ({username='slave'} at 127.0.99.132:58605)
I20240503 02:00:01.420163 6903 tablet_copy_service.cc:161] P 5dcc33cc621a4cbbae155f3230fb7c46: Beginning new tablet copy session on tablet f8d3d130f4a44904924f7ccf4316fc75 from peer 1d0e738c34af40fa9950c6f086619b3a at {username='slave'} at 127.0.99.132:58605: session id = 1d0e738c34af40fa9950c6f086619b3a-f8d3d130f4a44904924f7ccf4316fc75
I20240503 02:00:01.421380 6903 tablet_copy_source_session.cc:205] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:01.422223 17558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8d3d130f4a44904924f7ccf4316fc75. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.425729 17558 tablet_copy_client.cc:783] T f8d3d130f4a44904924f7ccf4316fc75 P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:01.426230 17558 tablet_copy_client.cc:647] T f8d3d130f4a44904924f7ccf4316fc75 P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:01.428489 17558 tablet_copy_client.cc:523] T f8d3d130f4a44904924f7ccf4316fc75 P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:01.430089 17558 tablet_bootstrap.cc:492] T f8d3d130f4a44904924f7ccf4316fc75 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 02:00:01.431546 16988 consensus_queue.cc:1035] T 9d44274085b9441c9f15676d842a0710 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:01.436208 15917 consensus_queue.cc:1035] T 9d44274085b9441c9f15676d842a0710 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:01.437978 9755 raft_consensus.cc:1203] T 8ca4e33409b74f359d3242248652d8bb P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:01.441332 17301 tablet_bootstrap.cc:492] T 76e8697ac8624e03a46bddae35f4c9fa P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:01.441664 17301 tablet_bootstrap.cc:492] T 76e8697ac8624e03a46bddae35f4c9fa P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap complete.
I20240503 02:00:01.441846 17301 ts_tablet_manager.cc:1367] T 76e8697ac8624e03a46bddae35f4c9fa P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.034s user 0.030s sys 0.000s
I20240503 02:00:01.442317 17301 raft_consensus.cc:348] T 76e8697ac8624e03a46bddae35f4c9fa P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:01.442440 17301 raft_consensus.cc:729] T 76e8697ac8624e03a46bddae35f4c9fa P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.442483 17558 tablet_bootstrap.cc:492] T f8d3d130f4a44904924f7ccf4316fc75 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:01.442598 17301 consensus_queue.cc:260] T 76e8697ac8624e03a46bddae35f4c9fa P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:01.442770 17558 tablet_bootstrap.cc:492] T f8d3d130f4a44904924f7ccf4316fc75 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap complete.
I20240503 02:00:01.442920 17558 ts_tablet_manager.cc:1367] T f8d3d130f4a44904924f7ccf4316fc75 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.001s
I20240503 02:00:01.442946 17301 ts_tablet_manager.cc:1398] T 76e8697ac8624e03a46bddae35f4c9fa P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 02:00:01.443434 17558 raft_consensus.cc:348] T f8d3d130f4a44904924f7ccf4316fc75 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:01.443562 17558 raft_consensus.cc:729] T f8d3d130f4a44904924f7ccf4316fc75 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.443717 17558 consensus_queue.cc:260] T f8d3d130f4a44904924f7ccf4316fc75 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:01.443989 17558 ts_tablet_manager.cc:1398] T f8d3d130f4a44904924f7ccf4316fc75 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:01.443982 7165 tablet_copy_service.cc:342] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Request end of tablet copy session db1a0000aaa346f79c0aa4b6c5105191-76e8697ac8624e03a46bddae35f4c9fa received from {username='slave'} at 127.0.99.136:40777
I20240503 02:00:01.444062 7165 tablet_copy_service.cc:433] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: ending tablet copy session db1a0000aaa346f79c0aa4b6c5105191-76e8697ac8624e03a46bddae35f4c9fa on tablet 76e8697ac8624e03a46bddae35f4c9fa with peer db1a0000aaa346f79c0aa4b6c5105191
I20240503 02:00:01.445677 6903 tablet_copy_service.cc:342] P 5dcc33cc621a4cbbae155f3230fb7c46: Request end of tablet copy session 1d0e738c34af40fa9950c6f086619b3a-f8d3d130f4a44904924f7ccf4316fc75 received from {username='slave'} at 127.0.99.132:58605
I20240503 02:00:01.445784 6903 tablet_copy_service.cc:433] P 5dcc33cc621a4cbbae155f3230fb7c46: ending tablet copy session 1d0e738c34af40fa9950c6f086619b3a-f8d3d130f4a44904924f7ccf4316fc75 on tablet f8d3d130f4a44904924f7ccf4316fc75 with peer 1d0e738c34af40fa9950c6f086619b3a
I20240503 02:00:01.460891 17415 ts_tablet_manager.cc:902] T b3ce9de18bda410aa204bf7c1c32e1b7 P 5dcc33cc621a4cbbae155f3230fb7c46: Initiating tablet copy from peer db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 02:00:01.461552 17415 tablet_copy_client.cc:308] T b3ce9de18bda410aa204bf7c1c32e1b7 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:42053
I20240503 02:00:01.462028 9777 tablet_copy_service.cc:140] P db1a0000aaa346f79c0aa4b6c5105191: Received BeginTabletCopySession request for tablet b3ce9de18bda410aa204bf7c1c32e1b7 from peer 5dcc33cc621a4cbbae155f3230fb7c46 ({username='slave'} at 127.0.99.129:48571)
I20240503 02:00:01.462136 9777 tablet_copy_service.cc:161] P db1a0000aaa346f79c0aa4b6c5105191: Beginning new tablet copy session on tablet b3ce9de18bda410aa204bf7c1c32e1b7 from peer 5dcc33cc621a4cbbae155f3230fb7c46 at {username='slave'} at 127.0.99.129:48571: session id = 5dcc33cc621a4cbbae155f3230fb7c46-b3ce9de18bda410aa204bf7c1c32e1b7
I20240503 02:00:01.463266 9777 tablet_copy_source_session.cc:205] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:01.464274 17415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3ce9de18bda410aa204bf7c1c32e1b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.464303 7014 consensus_queue.cc:237] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
W20240503 02:00:01.466404 6950 consensus_peers.cc:485] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): Couldn't send request to peer 5dcc33cc621a4cbbae155f3230fb7c46. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e20b869a89e44f1b0ba854163c31f06. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:01.466687 17415 tablet_copy_client.cc:783] T b3ce9de18bda410aa204bf7c1c32e1b7 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:01.467170 17415 tablet_copy_client.cc:647] T b3ce9de18bda410aa204bf7c1c32e1b7 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:01.467291 7145 raft_consensus.cc:1261] T 5e20b869a89e44f1b0ba854163c31f06 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:01.467466 9363 raft_consensus.cc:1261] T 5e20b869a89e44f1b0ba854163c31f06 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:01.467722 16845 consensus_queue.cc:1035] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:01.468078 17511 consensus_queue.cc:1035] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:01.468951 17415 tablet_copy_client.cc:523] T b3ce9de18bda410aa204bf7c1c32e1b7 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:01.470419 17415 tablet_bootstrap.cc:492] T b3ce9de18bda410aa204bf7c1c32e1b7 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 02:00:01.470979 16845 raft_consensus.cc:2932] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:01.473779 7145 raft_consensus.cc:2932] T 5e20b869a89e44f1b0ba854163c31f06 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:01.473840 9363 raft_consensus.cc:2932] T 5e20b869a89e44f1b0ba854163c31f06 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:01.476434 6734 catalog_manager.cc:5484] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:01.477818 17590 ts_tablet_manager.cc:902] T e811662836164c3fb29a1b12ace7eb41 P 12e8c683db054df3b75c6285b98a5ad8: Initiating tablet copy from peer 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 02:00:01.478929 17590 tablet_copy_client.cc:308] T e811662836164c3fb29a1b12ace7eb41 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36271
I20240503 02:00:01.479460 7295 tablet_copy_service.cc:140] P 1d0e738c34af40fa9950c6f086619b3a: Received BeginTabletCopySession request for tablet e811662836164c3fb29a1b12ace7eb41 from peer 12e8c683db054df3b75c6285b98a5ad8 ({username='slave'} at 127.0.99.133:37177)
I20240503 02:00:01.479571 7295 tablet_copy_service.cc:161] P 1d0e738c34af40fa9950c6f086619b3a: Beginning new tablet copy session on tablet e811662836164c3fb29a1b12ace7eb41 from peer 12e8c683db054df3b75c6285b98a5ad8 at {username='slave'} at 127.0.99.133:37177: session id = 12e8c683db054df3b75c6285b98a5ad8-e811662836164c3fb29a1b12ace7eb41
I20240503 02:00:01.480571 7295 tablet_copy_source_session.cc:205] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:01.482285 17415 tablet_bootstrap.cc:492] T b3ce9de18bda410aa204bf7c1c32e1b7 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:01.482394 17590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e811662836164c3fb29a1b12ace7eb41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.482594 17415 tablet_bootstrap.cc:492] T b3ce9de18bda410aa204bf7c1c32e1b7 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap complete.
I20240503 02:00:01.482774 17415 ts_tablet_manager.cc:1367] T b3ce9de18bda410aa204bf7c1c32e1b7 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20240503 02:00:01.483450 17415 raft_consensus.cc:348] T b3ce9de18bda410aa204bf7c1c32e1b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:01.483541 17415 raft_consensus.cc:729] T b3ce9de18bda410aa204bf7c1c32e1b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.483670 17415 consensus_queue.cc:260] T b3ce9de18bda410aa204bf7c1c32e1b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:01.483937 17415 ts_tablet_manager.cc:1398] T b3ce9de18bda410aa204bf7c1c32e1b7 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:01.485339 9777 tablet_copy_service.cc:342] P db1a0000aaa346f79c0aa4b6c5105191: Request end of tablet copy session 5dcc33cc621a4cbbae155f3230fb7c46-b3ce9de18bda410aa204bf7c1c32e1b7 received from {username='slave'} at 127.0.99.129:48571
I20240503 02:00:01.485414 9777 tablet_copy_service.cc:433] P db1a0000aaa346f79c0aa4b6c5105191: ending tablet copy session 5dcc33cc621a4cbbae155f3230fb7c46-b3ce9de18bda410aa204bf7c1c32e1b7 on tablet b3ce9de18bda410aa204bf7c1c32e1b7 with peer 5dcc33cc621a4cbbae155f3230fb7c46
I20240503 02:00:01.485636 17590 tablet_copy_client.cc:783] T e811662836164c3fb29a1b12ace7eb41 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:01.486400 17590 tablet_copy_client.cc:647] T e811662836164c3fb29a1b12ace7eb41 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:01.488217 6741 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:01.489151 17590 tablet_copy_client.cc:523] T e811662836164c3fb29a1b12ace7eb41 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:01.490869 17590 tablet_bootstrap.cc:492] T e811662836164c3fb29a1b12ace7eb41 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 02:00:01.497617 6882 raft_consensus.cc:1261] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:01.497665 9493 raft_consensus.cc:1261] T 4da3d94fdd1c42b2bc9cb08216bb84dc P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:01.498119 15917 consensus_queue.cc:1035] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:01.498275 16988 consensus_queue.cc:1035] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:01.501677 6735 catalog_manager.cc:5576] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943) does not have the latest schema for tablet 4da3d94fdd1c42b2bc9cb08216bb84dc (table rebalancer_dynamic_table [id=3bbac19b98664efeaf8f0aafd4e7655f]). Expected version 1 got 0
I20240503 02:00:01.502370 6734 catalog_manager.cc:5576] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471) does not have the latest schema for tablet 4da3d94fdd1c42b2bc9cb08216bb84dc (table rebalancer_dynamic_table [id=3bbac19b98664efeaf8f0aafd4e7655f]). Expected version 1 got 0
I20240503 02:00:01.503001 17274 tablet.cc:1722] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:01.503541 17274 tablet.cc:1620] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 02:00:01.507422 6735 catalog_manager.cc:5576] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471) does not have the latest schema for tablet 4da3d94fdd1c42b2bc9cb08216bb84dc (table rebalancer_dynamic_table [id=3bbac19b98664efeaf8f0aafd4e7655f]). Expected version 1 got 0
I20240503 02:00:01.514115 6734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=3bbac19b98664efeaf8f0aafd4e7655f] alter complete (version 1)
I20240503 02:00:01.522076 17611 tablet.cc:1722] T 4da3d94fdd1c42b2bc9cb08216bb84dc P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:01.522574 17611 tablet.cc:1620] T 4da3d94fdd1c42b2bc9cb08216bb84dc P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 02:00:01.523532 17610 ts_tablet_manager.cc:902] T d80398c9a264430197262bb42241d58d P bb5364d730cb4489a0bfc6c270b7450a: Initiating tablet copy from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 02:00:01.523854 6881 raft_consensus.cc:1203] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:01.524338 17610 tablet_copy_client.cc:308] T d80398c9a264430197262bb42241d58d P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41979
I20240503 02:00:01.524863 7165 tablet_copy_service.cc:140] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Received BeginTabletCopySession request for tablet d80398c9a264430197262bb42241d58d from peer bb5364d730cb4489a0bfc6c270b7450a ({username='slave'} at 127.0.99.134:59853)
I20240503 02:00:01.524987 7165 tablet_copy_service.cc:161] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Beginning new tablet copy session on tablet d80398c9a264430197262bb42241d58d from peer bb5364d730cb4489a0bfc6c270b7450a at {username='slave'} at 127.0.99.134:59853: session id = bb5364d730cb4489a0bfc6c270b7450a-d80398c9a264430197262bb42241d58d
I20240503 02:00:01.525902 7165 tablet_copy_source_session.cc:205] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:01.527223 17610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d80398c9a264430197262bb42241d58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.532420 17610 tablet_copy_client.cc:783] T d80398c9a264430197262bb42241d58d P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:01.532914 17610 tablet_copy_client.cc:647] T d80398c9a264430197262bb42241d58d P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:01.534627 17610 tablet_copy_client.cc:523] T d80398c9a264430197262bb42241d58d P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:01.536505 17610 tablet_bootstrap.cc:492] T d80398c9a264430197262bb42241d58d P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 02:00:01.537420 17590 tablet_bootstrap.cc:492] T e811662836164c3fb29a1b12ace7eb41 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:01.537828 17590 tablet_bootstrap.cc:492] T e811662836164c3fb29a1b12ace7eb41 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap complete.
I20240503 02:00:01.538061 17590 ts_tablet_manager.cc:1367] T e811662836164c3fb29a1b12ace7eb41 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.047s user 0.029s sys 0.000s
I20240503 02:00:01.538635 17590 raft_consensus.cc:348] T e811662836164c3fb29a1b12ace7eb41 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:01.538851 17590 raft_consensus.cc:729] T e811662836164c3fb29a1b12ace7eb41 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.539105 17590 consensus_queue.cc:260] T e811662836164c3fb29a1b12ace7eb41 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:01.541030 17590 ts_tablet_manager.cc:1398] T e811662836164c3fb29a1b12ace7eb41 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:01.541738 7295 tablet_copy_service.cc:342] P 1d0e738c34af40fa9950c6f086619b3a: Request end of tablet copy session 12e8c683db054df3b75c6285b98a5ad8-e811662836164c3fb29a1b12ace7eb41 received from {username='slave'} at 127.0.99.133:37177
I20240503 02:00:01.541829 7295 tablet_copy_service.cc:433] P 1d0e738c34af40fa9950c6f086619b3a: ending tablet copy session 12e8c683db054df3b75c6285b98a5ad8-e811662836164c3fb29a1b12ace7eb41 on tablet e811662836164c3fb29a1b12ace7eb41 with peer 12e8c683db054df3b75c6285b98a5ad8
I20240503 02:00:01.543370 17610 tablet_bootstrap.cc:492] T d80398c9a264430197262bb42241d58d P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:01.543295 17616 tablet.cc:1722] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:01.543745 17610 tablet_bootstrap.cc:492] T d80398c9a264430197262bb42241d58d P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap complete.
I20240503 02:00:01.544090 17616 tablet.cc:1620] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 5dcc33cc621a4cbbae155f3230fb7c46: MemRowSet was empty: no flush needed.
I20240503 02:00:01.544093 17610 ts_tablet_manager.cc:1367] T d80398c9a264430197262bb42241d58d P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20240503 02:00:01.544663 17610 raft_consensus.cc:348] T d80398c9a264430197262bb42241d58d P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:01.544871 17610 raft_consensus.cc:729] T d80398c9a264430197262bb42241d58d P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.545135 17610 consensus_queue.cc:260] T d80398c9a264430197262bb42241d58d P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:01.546422 17610 ts_tablet_manager.cc:1398] T d80398c9a264430197262bb42241d58d P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:01.547562 7165 tablet_copy_service.cc:342] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Request end of tablet copy session bb5364d730cb4489a0bfc6c270b7450a-d80398c9a264430197262bb42241d58d received from {username='slave'} at 127.0.99.134:59853
I20240503 02:00:01.547679 7165 tablet_copy_service.cc:433] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: ending tablet copy session bb5364d730cb4489a0bfc6c270b7450a-d80398c9a264430197262bb42241d58d on tablet d80398c9a264430197262bb42241d58d with peer bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:01.548661 7145 raft_consensus.cc:1203] T 9e21d082e40540d4bb908fc9e90055c9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:01.550542 6735 catalog_manager.cc:5576] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943) does not have the latest schema for tablet 4da3d94fdd1c42b2bc9cb08216bb84dc (table rebalancer_dynamic_table [id=3bbac19b98664efeaf8f0aafd4e7655f]). Expected version 1 got 0
I20240503 02:00:01.554133 7014 raft_consensus.cc:1203] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:01.558605 9757 raft_consensus.cc:1203] T 4bd8226227c348d5a5d4d8ff28959d27 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 02:00:01.574076 16962 consensus_queue.cc:1035] T 5a1b7f0005e54894b0bc28c3b988a14a P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:01.578606 16962 consensus_queue.cc:1035] T 5a1b7f0005e54894b0bc28c3b988a14a P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:01.580489 7145 raft_consensus.cc:1203] T 0bbbd17c78bc4e329a3a7c7008a34d86 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:01.611236 17650 ts_tablet_manager.cc:902] T 22b6ec13c4e04d6cbade150f0c644e52 P b7062fbbc2b54fff9e99ec4ea5562c76: Initiating tablet copy from peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:01.611862 17650 tablet_copy_client.cc:308] T 22b6ec13c4e04d6cbade150f0c644e52 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:37387
I20240503 02:00:01.612443 9383 tablet_copy_service.cc:140] P 12e8c683db054df3b75c6285b98a5ad8: Received BeginTabletCopySession request for tablet 22b6ec13c4e04d6cbade150f0c644e52 from peer b7062fbbc2b54fff9e99ec4ea5562c76 ({username='slave'} at 127.0.99.135:33951)
I20240503 02:00:01.612555 9383 tablet_copy_service.cc:161] P 12e8c683db054df3b75c6285b98a5ad8: Beginning new tablet copy session on tablet 22b6ec13c4e04d6cbade150f0c644e52 from peer b7062fbbc2b54fff9e99ec4ea5562c76 at {username='slave'} at 127.0.99.135:33951: session id = b7062fbbc2b54fff9e99ec4ea5562c76-22b6ec13c4e04d6cbade150f0c644e52
I20240503 02:00:01.613564 9383 tablet_copy_source_session.cc:205] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:01.622776 17650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b6ec13c4e04d6cbade150f0c644e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.619264 6732 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:01.628813 17650 tablet_copy_client.cc:783] T 22b6ec13c4e04d6cbade150f0c644e52 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:01.629428 17650 tablet_copy_client.cc:647] T 22b6ec13c4e04d6cbade150f0c644e52 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:01.632232 17274 tablet.cc:1722] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:01.632645 17274 tablet.cc:1620] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 02:00:01.633327 17611 tablet.cc:1722] T 4da3d94fdd1c42b2bc9cb08216bb84dc P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:01.633394 17616 tablet.cc:1722] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:01.633636 17611 tablet.cc:1620] T 4da3d94fdd1c42b2bc9cb08216bb84dc P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 02:00:01.633714 17616 tablet.cc:1620] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 5dcc33cc621a4cbbae155f3230fb7c46: MemRowSet was empty: no flush needed.
I20240503 02:00:01.635352 17650 tablet_copy_client.cc:523] T 22b6ec13c4e04d6cbade150f0c644e52 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:01.636672 17650 tablet_bootstrap.cc:492] T 22b6ec13c4e04d6cbade150f0c644e52 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 02:00:01.640622 17650 tablet_bootstrap.cc:492] T 22b6ec13c4e04d6cbade150f0c644e52 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:01.641119 17650 tablet_bootstrap.cc:492] T 22b6ec13c4e04d6cbade150f0c644e52 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap complete.
I20240503 02:00:01.641510 17650 ts_tablet_manager.cc:1367] T 22b6ec13c4e04d6cbade150f0c644e52 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.005s user 0.006s sys 0.000s
I20240503 02:00:01.642177 17650 raft_consensus.cc:348] T 22b6ec13c4e04d6cbade150f0c644e52 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:01.642318 17650 raft_consensus.cc:729] T 22b6ec13c4e04d6cbade150f0c644e52 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.642477 17650 consensus_queue.cc:260] T 22b6ec13c4e04d6cbade150f0c644e52 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:01.643108 17650 ts_tablet_manager.cc:1398] T 22b6ec13c4e04d6cbade150f0c644e52 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:01.644361 9383 tablet_copy_service.cc:342] P 12e8c683db054df3b75c6285b98a5ad8: Request end of tablet copy session b7062fbbc2b54fff9e99ec4ea5562c76-22b6ec13c4e04d6cbade150f0c644e52 received from {username='slave'} at 127.0.99.135:33951
I20240503 02:00:01.644456 9383 tablet_copy_service.cc:433] P 12e8c683db054df3b75c6285b98a5ad8: ending tablet copy session b7062fbbc2b54fff9e99ec4ea5562c76-22b6ec13c4e04d6cbade150f0c644e52 on tablet 22b6ec13c4e04d6cbade150f0c644e52 with peer b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:01.651895 6737 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:01.655787 6738 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=3bbac19b98664efeaf8f0aafd4e7655f] alter complete (version 2)
I20240503 02:00:01.670861 9604 tablet_service.cc:1467] Processing CreateTablet for tablet 9b23d84ca6e343b4b550c3025d2b1f12 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=52ec69f7df16443e8705f5eb8083b0ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:01.670980 9605 tablet_service.cc:1467] Processing CreateTablet for tablet c9173280a825443db6088e7141217b29 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=52ec69f7df16443e8705f5eb8083b0ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:01.671226 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b23d84ca6e343b4b550c3025d2b1f12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.671352 9605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9173280a825443db6088e7141217b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.671305 9342 tablet_service.cc:1467] Processing CreateTablet for tablet c9173280a825443db6088e7141217b29 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=52ec69f7df16443e8705f5eb8083b0ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:01.671567 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9173280a825443db6088e7141217b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.673563 17656 tablet_bootstrap.cc:492] T c9173280a825443db6088e7141217b29 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 02:00:01.673799 17122 tablet_bootstrap.cc:492] T c9173280a825443db6088e7141217b29 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 02:00:01.674854 17656 tablet_bootstrap.cc:654] T c9173280a825443db6088e7141217b29 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:01.674855 17122 tablet_bootstrap.cc:654] T c9173280a825443db6088e7141217b29 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:01.676081 7254 tablet_service.cc:1467] Processing CreateTablet for tablet c9173280a825443db6088e7141217b29 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=52ec69f7df16443e8705f5eb8083b0ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:01.676370 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9173280a825443db6088e7141217b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.677240 7256 tablet_service.cc:1467] Processing CreateTablet for tablet 9b23d84ca6e343b4b550c3025d2b1f12 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=52ec69f7df16443e8705f5eb8083b0ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:01.677486 7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b23d84ca6e343b4b550c3025d2b1f12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.679755 6992 tablet_service.cc:1467] Processing CreateTablet for tablet 9b23d84ca6e343b4b550c3025d2b1f12 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=52ec69f7df16443e8705f5eb8083b0ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:01.679893 7255 tablet_service.cc:1467] Processing CreateTablet for tablet 9cc9a609452f4f4aad600381968b9227 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=52ec69f7df16443e8705f5eb8083b0ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:01.680058 6992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b23d84ca6e343b4b550c3025d2b1f12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.680095 7255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cc9a609452f4f4aad600381968b9227. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.682796 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 9cc9a609452f4f4aad600381968b9227 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=52ec69f7df16443e8705f5eb8083b0ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:01.683111 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cc9a609452f4f4aad600381968b9227. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.684203 17659 tablet_bootstrap.cc:492] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 02:00:01.684278 6863 tablet_service.cc:1467] Processing CreateTablet for tablet 9cc9a609452f4f4aad600381968b9227 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=52ec69f7df16443e8705f5eb8083b0ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:01.684557 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cc9a609452f4f4aad600381968b9227. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.685528 17659 tablet_bootstrap.cc:654] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:01.687299 17659 tablet_bootstrap.cc:492] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 02:00:01.687817 17659 ts_tablet_manager.cc:1367] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:00:01.688660 17659 raft_consensus.cc:348] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:01.688822 17659 raft_consensus.cc:374] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:01.688863 17659 raft_consensus.cc:729] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.689014 17659 consensus_queue.cc:260] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:01.689426 17659 ts_tablet_manager.cc:1398] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:01.689719 17659 tablet_bootstrap.cc:492] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 02:00:01.691473 17659 tablet_bootstrap.cc:654] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:01.691643 17495 tablet_bootstrap.cc:492] T 9cc9a609452f4f4aad600381968b9227 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 02:00:01.692121 17660 tablet_bootstrap.cc:492] T 9b23d84ca6e343b4b550c3025d2b1f12 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 02:00:01.692917 17122 tablet_bootstrap.cc:492] T c9173280a825443db6088e7141217b29 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 02:00:01.693032 17122 ts_tablet_manager.cc:1367] T c9173280a825443db6088e7141217b29 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.002s
I20240503 02:00:01.693300 17495 tablet_bootstrap.cc:654] T 9cc9a609452f4f4aad600381968b9227 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:01.693545 17122 raft_consensus.cc:348] T c9173280a825443db6088e7141217b29 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:01.693665 17122 raft_consensus.cc:374] T c9173280a825443db6088e7141217b29 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:01.693698 17122 raft_consensus.cc:729] T c9173280a825443db6088e7141217b29 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.693825 17122 consensus_queue.cc:260] T c9173280a825443db6088e7141217b29 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:01.694114 17122 ts_tablet_manager.cc:1398] T c9173280a825443db6088e7141217b29 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:01.694414 17122 tablet_bootstrap.cc:492] T 9cc9a609452f4f4aad600381968b9227 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 02:00:01.696653 17495 tablet_bootstrap.cc:492] T 9cc9a609452f4f4aad600381968b9227 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 02:00:01.696766 17495 ts_tablet_manager.cc:1367] T 9cc9a609452f4f4aad600381968b9227 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:01.697309 17495 raft_consensus.cc:348] T 9cc9a609452f4f4aad600381968b9227 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:01.697455 17495 raft_consensus.cc:374] T 9cc9a609452f4f4aad600381968b9227 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:01.697499 17495 raft_consensus.cc:729] T 9cc9a609452f4f4aad600381968b9227 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.697677 17495 consensus_queue.cc:260] T 9cc9a609452f4f4aad600381968b9227 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:01.697971 17495 ts_tablet_manager.cc:1398] T 9cc9a609452f4f4aad600381968b9227 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:01.698446 17660 tablet_bootstrap.cc:654] T 9b23d84ca6e343b4b550c3025d2b1f12 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:01.699810 17122 tablet_bootstrap.cc:654] T 9cc9a609452f4f4aad600381968b9227 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:01.701435 17659 tablet_bootstrap.cc:492] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 02:00:01.702344 17656 tablet_bootstrap.cc:492] T c9173280a825443db6088e7141217b29 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 02:00:01.702471 17656 ts_tablet_manager.cc:1367] T c9173280a825443db6088e7141217b29 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20240503 02:00:01.702504 17660 tablet_bootstrap.cc:492] T 9b23d84ca6e343b4b550c3025d2b1f12 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 02:00:01.702634 17660 ts_tablet_manager.cc:1367] T 9b23d84ca6e343b4b550c3025d2b1f12 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:00:01.703151 17660 raft_consensus.cc:348] T 9b23d84ca6e343b4b550c3025d2b1f12 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:01.703342 17660 raft_consensus.cc:374] T 9b23d84ca6e343b4b550c3025d2b1f12 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:01.703322 17656 raft_consensus.cc:348] T c9173280a825443db6088e7141217b29 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:01.703420 17660 raft_consensus.cc:729] T 9b23d84ca6e343b4b550c3025d2b1f12 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.703435 17656 raft_consensus.cc:374] T c9173280a825443db6088e7141217b29 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:01.703470 17656 raft_consensus.cc:729] T c9173280a825443db6088e7141217b29 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.703573 17656 consensus_queue.cc:260] T c9173280a825443db6088e7141217b29 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:01.703565 17660 consensus_queue.cc:260] T 9b23d84ca6e343b4b550c3025d2b1f12 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:01.703846 17656 ts_tablet_manager.cc:1398] T c9173280a825443db6088e7141217b29 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:01.704001 17656 tablet_bootstrap.cc:492] T 9b23d84ca6e343b4b550c3025d2b1f12 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 02:00:01.704890 17660 ts_tablet_manager.cc:1398] T 9b23d84ca6e343b4b550c3025d2b1f12 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:01.704994 17656 tablet_bootstrap.cc:654] T 9b23d84ca6e343b4b550c3025d2b1f12 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:01.706701 17122 tablet_bootstrap.cc:492] T 9cc9a609452f4f4aad600381968b9227 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 02:00:01.706928 17122 ts_tablet_manager.cc:1367] T 9cc9a609452f4f4aad600381968b9227 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:00:01.707710 17122 raft_consensus.cc:348] T 9cc9a609452f4f4aad600381968b9227 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:01.708531 17122 raft_consensus.cc:374] T 9cc9a609452f4f4aad600381968b9227 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:01.709008 17122 raft_consensus.cc:729] T 9cc9a609452f4f4aad600381968b9227 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.709359 17122 consensus_queue.cc:260] T 9cc9a609452f4f4aad600381968b9227 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:01.709790 17122 ts_tablet_manager.cc:1398] T 9cc9a609452f4f4aad600381968b9227 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:01.710004 17659 ts_tablet_manager.cc:1367] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20240503 02:00:01.710917 17659 raft_consensus.cc:348] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:01.711125 17659 raft_consensus.cc:374] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:01.711205 17659 raft_consensus.cc:729] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.711409 17659 consensus_queue.cc:260] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:01.712916 17659 ts_tablet_manager.cc:1398] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:00:01.713164 17659 tablet_bootstrap.cc:492] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 02:00:01.715637 17656 tablet_bootstrap.cc:492] T 9b23d84ca6e343b4b550c3025d2b1f12 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 02:00:01.715744 17656 ts_tablet_manager.cc:1367] T 9b23d84ca6e343b4b550c3025d2b1f12 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:00:01.716266 17656 raft_consensus.cc:348] T 9b23d84ca6e343b4b550c3025d2b1f12 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:01.716383 17656 raft_consensus.cc:374] T 9b23d84ca6e343b4b550c3025d2b1f12 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:01.716437 17656 raft_consensus.cc:729] T 9b23d84ca6e343b4b550c3025d2b1f12 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.716558 17656 consensus_queue.cc:260] T 9b23d84ca6e343b4b550c3025d2b1f12 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:01.716861 17656 ts_tablet_manager.cc:1398] T 9b23d84ca6e343b4b550c3025d2b1f12 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:01.719101 17659 tablet_bootstrap.cc:654] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:01.721105 17590 ts_tablet_manager.cc:902] T fd29f3b059874b019153b64ef8aaaa48 P 12e8c683db054df3b75c6285b98a5ad8: Initiating tablet copy from peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 02:00:01.721684 17590 tablet_copy_client.cc:308] T fd29f3b059874b019153b64ef8aaaa48 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:42943
I20240503 02:00:01.722237 7254 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:01.722437 6863 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:01.722952 6903 tablet_copy_service.cc:140] P 5dcc33cc621a4cbbae155f3230fb7c46: Received BeginTabletCopySession request for tablet fd29f3b059874b019153b64ef8aaaa48 from peer 12e8c683db054df3b75c6285b98a5ad8 ({username='slave'} at 127.0.99.133:55147)
I20240503 02:00:01.723064 6903 tablet_copy_service.cc:161] P 5dcc33cc621a4cbbae155f3230fb7c46: Beginning new tablet copy session on tablet fd29f3b059874b019153b64ef8aaaa48 from peer 12e8c683db054df3b75c6285b98a5ad8 at {username='slave'} at 127.0.99.133:55147: session id = 12e8c683db054df3b75c6285b98a5ad8-fd29f3b059874b019153b64ef8aaaa48
I20240503 02:00:01.724066 6903 tablet_copy_source_session.cc:205] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:01.725528 17659 tablet_bootstrap.cc:492] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 02:00:01.725670 17659 ts_tablet_manager.cc:1367] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.003s
I20240503 02:00:01.725826 17590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd29f3b059874b019153b64ef8aaaa48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.726248 17659 raft_consensus.cc:348] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:01.726353 17659 raft_consensus.cc:374] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:01.726594 17659 raft_consensus.cc:729] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.726744 17659 consensus_queue.cc:260] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:01.726971 17659 ts_tablet_manager.cc:1398] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:01.727986 17590 tablet_copy_client.cc:783] T fd29f3b059874b019153b64ef8aaaa48 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:01.728273 17590 tablet_copy_client.cc:647] T fd29f3b059874b019153b64ef8aaaa48 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:01.729594 6882 raft_consensus.cc:1203] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:01.749212 9736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:00:01.749547 17687 ts_tablet_manager.cc:902] T 3e824fcc9aac449d9373655b304010e2 P 63331f455be049c1bdd6c23c59ac1bef: Initiating tablet copy from peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 02:00:01.750125 6992 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:00:01.750371 17687 tablet_copy_client.cc:308] T 3e824fcc9aac449d9373655b304010e2 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:43471
I20240503 02:00:01.751108 9514 tablet_copy_service.cc:140] P bb5364d730cb4489a0bfc6c270b7450a: Received BeginTabletCopySession request for tablet 3e824fcc9aac449d9373655b304010e2 from peer 63331f455be049c1bdd6c23c59ac1bef ({username='slave'} at 127.0.99.130:54311)
I20240503 02:00:01.751279 9514 tablet_copy_service.cc:161] P bb5364d730cb4489a0bfc6c270b7450a: Beginning new tablet copy session on tablet 3e824fcc9aac449d9373655b304010e2 from peer 63331f455be049c1bdd6c23c59ac1bef at {username='slave'} at 127.0.99.130:54311: session id = 63331f455be049c1bdd6c23c59ac1bef-3e824fcc9aac449d9373655b304010e2
I20240503 02:00:01.752588 9514 tablet_copy_source_session.cc:205] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:01.756143 17687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e824fcc9aac449d9373655b304010e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.758251 17687 tablet_copy_client.cc:783] T 3e824fcc9aac449d9373655b304010e2 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:01.758440 17687 tablet_copy_client.cc:647] T 3e824fcc9aac449d9373655b304010e2 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:01.759006 9342 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:00:01.762043 9472 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:01.766359 6732 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:01.774672 17616 tablet.cc:1722] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:01.774430 17274 tablet.cc:1722] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:01.776373 7014 raft_consensus.cc:1203] T b93fe374d2a9496a847480b7824a5937 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:01.781539 7124 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:00:01.783701 6737 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=3bbac19b98664efeaf8f0aafd4e7655f] alter complete (version 3)
I20240503 02:00:01.784086 17590 tablet_copy_client.cc:523] T fd29f3b059874b019153b64ef8aaaa48 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:01.787942 17590 tablet_bootstrap.cc:492] T fd29f3b059874b019153b64ef8aaaa48 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 02:00:01.791010 17611 tablet.cc:1722] T 4da3d94fdd1c42b2bc9cb08216bb84dc P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:01.798652 16988 raft_consensus.cc:482] T 9cc9a609452f4f4aad600381968b9227 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:01.798794 16988 raft_consensus.cc:504] T 9cc9a609452f4f4aad600381968b9227 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:01.799237 16988 leader_election.cc:288] T 9cc9a609452f4f4aad600381968b9227 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 02:00:01.799737 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cc9a609452f4f4aad600381968b9227" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 02:00:01.799891 7275 raft_consensus.cc:2454] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 02:00:01.800226 9297 leader_election.cc:302] T 9cc9a609452f4f4aad600381968b9227 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 1d0e738c34af40fa9950c6f086619b3a; no voters:
I20240503 02:00:01.800462 16988 raft_consensus.cc:2781] T 9cc9a609452f4f4aad600381968b9227 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:01.800516 16988 raft_consensus.cc:482] T 9cc9a609452f4f4aad600381968b9227 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:01.800551 16988 raft_consensus.cc:3037] T 9cc9a609452f4f4aad600381968b9227 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:01.801044 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cc9a609452f4f4aad600381968b9227" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 02:00:01.801177 6882 raft_consensus.cc:2454] T 9cc9a609452f4f4aad600381968b9227 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 02:00:01.801713 16988 raft_consensus.cc:504] T 9cc9a609452f4f4aad600381968b9227 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:01.802104 17692 ts_tablet_manager.cc:902] T e8cb37039be049cf9c4e53bbcfbd80d9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Initiating tablet copy from peer db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 02:00:01.803149 17692 tablet_copy_client.cc:308] T e8cb37039be049cf9c4e53bbcfbd80d9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:42053
I20240503 02:00:01.804291 9777 tablet_copy_service.cc:140] P db1a0000aaa346f79c0aa4b6c5105191: Received BeginTabletCopySession request for tablet e8cb37039be049cf9c4e53bbcfbd80d9 from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a ({username='slave'} at 127.0.99.131:54483)
I20240503 02:00:01.804409 9777 tablet_copy_service.cc:161] P db1a0000aaa346f79c0aa4b6c5105191: Beginning new tablet copy session on tablet e8cb37039be049cf9c4e53bbcfbd80d9 from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a at {username='slave'} at 127.0.99.131:54483: session id = c688f3df56ac4b6d9b0c30fa4cc8ca2a-e8cb37039be049cf9c4e53bbcfbd80d9
I20240503 02:00:01.804611 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cc9a609452f4f4aad600381968b9227" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 02:00:01.804765 6882 raft_consensus.cc:3037] T 9cc9a609452f4f4aad600381968b9227 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:01.805415 9777 tablet_copy_source_session.cc:205] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:01.805864 6882 raft_consensus.cc:2454] T 9cc9a609452f4f4aad600381968b9227 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 02:00:01.806393 9299 leader_election.cc:302] T 9cc9a609452f4f4aad600381968b9227 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 02:00:01.806718 17147 raft_consensus.cc:2781] T 9cc9a609452f4f4aad600381968b9227 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:01.806831 17147 raft_consensus.cc:686] T 9cc9a609452f4f4aad600381968b9227 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Running, Role: LEADER
I20240503 02:00:01.806960 17147 consensus_queue.cc:237] T 9cc9a609452f4f4aad600381968b9227 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:01.809177 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cc9a609452f4f4aad600381968b9227" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 02:00:01.807548 17692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8cb37039be049cf9c4e53bbcfbd80d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.809315 7275 raft_consensus.cc:3037] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:01.810099 6737 catalog_manager.cc:5484] T 9cc9a609452f4f4aad600381968b9227 P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:01.810501 7275 raft_consensus.cc:2454] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 02:00:01.812022 17692 tablet_copy_client.cc:783] T e8cb37039be049cf9c4e53bbcfbd80d9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:01.812837 16988 leader_election.cc:288] T 9cc9a609452f4f4aad600381968b9227 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Requested vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 02:00:01.814097 17692 tablet_copy_client.cc:647] T e8cb37039be049cf9c4e53bbcfbd80d9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:01.817428 17557 raft_consensus.cc:482] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:01.817535 17557 raft_consensus.cc:504] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:01.818020 17557 leader_election.cc:288] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 02:00:01.818470 9624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b23d84ca6e343b4b550c3025d2b1f12" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 02:00:01.818647 9624 raft_consensus.cc:2454] T 9b23d84ca6e343b4b550c3025d2b1f12 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 02:00:01.818639 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b23d84ca6e343b4b550c3025d2b1f12" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 02:00:01.818785 7014 raft_consensus.cc:2454] T 9b23d84ca6e343b4b550c3025d2b1f12 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 02:00:01.819044 7211 leader_election.cc:302] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 02:00:01.819293 17557 raft_consensus.cc:2781] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:01.819340 17557 raft_consensus.cc:482] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:01.819372 17557 raft_consensus.cc:3037] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:01.820462 17557 raft_consensus.cc:504] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:01.820843 17557 leader_election.cc:288] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 02:00:01.821954 9624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b23d84ca6e343b4b550c3025d2b1f12" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 02:00:01.822090 9624 raft_consensus.cc:3037] T 9b23d84ca6e343b4b550c3025d2b1f12 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:01.822336 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b23d84ca6e343b4b550c3025d2b1f12" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 02:00:01.822468 7014 raft_consensus.cc:3037] T 9b23d84ca6e343b4b550c3025d2b1f12 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:01.823230 9624 raft_consensus.cc:2454] T 9b23d84ca6e343b4b550c3025d2b1f12 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 02:00:01.823658 7211 leader_election.cc:302] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 02:00:01.823886 17557 raft_consensus.cc:2781] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:01.823998 17557 raft_consensus.cc:686] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 02:00:01.823953 7014 raft_consensus.cc:2454] T 9b23d84ca6e343b4b550c3025d2b1f12 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 02:00:01.824127 17557 consensus_queue.cc:237] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:01.827015 17687 tablet_copy_client.cc:523] T 3e824fcc9aac449d9373655b304010e2 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:01.827190 6737 catalog_manager.cc:5484] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:01.832415 17687 tablet_bootstrap.cc:492] T 3e824fcc9aac449d9373655b304010e2 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 02:00:01.832695 17590 tablet_bootstrap.cc:492] T fd29f3b059874b019153b64ef8aaaa48 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:01.833050 17590 tablet_bootstrap.cc:492] T fd29f3b059874b019153b64ef8aaaa48 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap complete.
I20240503 02:00:01.833312 17590 ts_tablet_manager.cc:1367] T fd29f3b059874b019153b64ef8aaaa48 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.000s
I20240503 02:00:01.834045 17590 raft_consensus.cc:348] T fd29f3b059874b019153b64ef8aaaa48 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:01.834267 17590 raft_consensus.cc:729] T fd29f3b059874b019153b64ef8aaaa48 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.834489 17590 consensus_queue.cc:260] T fd29f3b059874b019153b64ef8aaaa48 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:01.838112 9605 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:00:01.840286 17590 ts_tablet_manager.cc:1398] T fd29f3b059874b019153b64ef8aaaa48 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 02:00:01.840596 17692 tablet_copy_client.cc:523] T e8cb37039be049cf9c4e53bbcfbd80d9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:01.842293 17692 tablet_bootstrap.cc:492] T e8cb37039be049cf9c4e53bbcfbd80d9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 02:00:01.843909 16102 raft_consensus.cc:482] T c9173280a825443db6088e7141217b29 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:01.844010 16102 raft_consensus.cc:504] T c9173280a825443db6088e7141217b29 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:01.844417 16102 leader_election.cc:288] T c9173280a825443db6088e7141217b29 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 02:00:01.845060 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9173280a825443db6088e7141217b29" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 02:00:01.845202 9363 raft_consensus.cc:2454] T c9173280a825443db6088e7141217b29 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 0.
I20240503 02:00:01.845611 9561 leader_election.cc:302] T c9173280a825443db6088e7141217b29 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 02:00:01.845935 16102 raft_consensus.cc:2781] T c9173280a825443db6088e7141217b29 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:01.845991 16102 raft_consensus.cc:482] T c9173280a825443db6088e7141217b29 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:01.846030 16102 raft_consensus.cc:3037] T c9173280a825443db6088e7141217b29 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:01.846757 6903 tablet_copy_service.cc:342] P 5dcc33cc621a4cbbae155f3230fb7c46: Request end of tablet copy session 12e8c683db054df3b75c6285b98a5ad8-fd29f3b059874b019153b64ef8aaaa48 received from {username='slave'} at 127.0.99.133:55147
I20240503 02:00:01.846861 6903 tablet_copy_service.cc:433] P 5dcc33cc621a4cbbae155f3230fb7c46: ending tablet copy session 12e8c683db054df3b75c6285b98a5ad8-fd29f3b059874b019153b64ef8aaaa48 on tablet fd29f3b059874b019153b64ef8aaaa48 with peer 12e8c683db054df3b75c6285b98a5ad8
I20240503 02:00:01.847983 16102 raft_consensus.cc:504] T c9173280a825443db6088e7141217b29 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:01.848366 16102 leader_election.cc:288] T c9173280a825443db6088e7141217b29 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Requested vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 02:00:01.848873 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9173280a825443db6088e7141217b29" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 02:00:01.849007 7274 raft_consensus.cc:3037] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:01.849261 6882 raft_consensus.cc:1203] T b3ce9de18bda410aa204bf7c1c32e1b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:01.849349 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9173280a825443db6088e7141217b29" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 02:00:01.849457 9363 raft_consensus.cc:3037] T c9173280a825443db6088e7141217b29 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:01.850116 7274 raft_consensus.cc:2454] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 02:00:01.850466 9363 raft_consensus.cc:2454] T c9173280a825443db6088e7141217b29 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1.
I20240503 02:00:01.850805 9560 leader_election.cc:302] T c9173280a825443db6088e7141217b29 P b7062fbbc2b54fff9e99ec4ea5562c76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 02:00:01.852094 7274 raft_consensus.cc:1203] T f8d3d130f4a44904924f7ccf4316fc75 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:01.853000 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9173280a825443db6088e7141217b29" candidate_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 02:00:01.853752 7275 raft_consensus.cc:2362] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate b7062fbbc2b54fff9e99ec4ea5562c76 in term 1. Re-sending same reply.
I20240503 02:00:01.853806 16102 raft_consensus.cc:2781] T c9173280a825443db6088e7141217b29 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:01.853912 16102 raft_consensus.cc:686] T c9173280a825443db6088e7141217b29 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 LEADER]: Becoming Leader. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Running, Role: LEADER
I20240503 02:00:01.854171 16102 consensus_queue.cc:237] T c9173280a825443db6088e7141217b29 P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:01.856978 6737 catalog_manager.cc:5484] T c9173280a825443db6088e7141217b29 P b7062fbbc2b54fff9e99ec4ea5562c76 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135). New cstate: current_term: 1 leader_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:01.861109 17692 tablet_bootstrap.cc:492] T e8cb37039be049cf9c4e53bbcfbd80d9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:01.861407 17692 tablet_bootstrap.cc:492] T e8cb37039be049cf9c4e53bbcfbd80d9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap complete.
I20240503 02:00:01.861578 17692 ts_tablet_manager.cc:1367] T e8cb37039be049cf9c4e53bbcfbd80d9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20240503 02:00:01.862085 17692 raft_consensus.cc:348] T e8cb37039be049cf9c4e53bbcfbd80d9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:01.862211 17692 raft_consensus.cc:729] T e8cb37039be049cf9c4e53bbcfbd80d9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.862461 17692 consensus_queue.cc:260] T e8cb37039be049cf9c4e53bbcfbd80d9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:01.865315 17558 ts_tablet_manager.cc:902] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 1d0e738c34af40fa9950c6f086619b3a: Initiating tablet copy from peer 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 02:00:01.865365 17692 ts_tablet_manager.cc:1398] T e8cb37039be049cf9c4e53bbcfbd80d9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:00:01.866003 17558 tablet_copy_client.cc:308] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:37231
I20240503 02:00:01.866227 9777 tablet_copy_service.cc:342] P db1a0000aaa346f79c0aa4b6c5105191: Request end of tablet copy session c688f3df56ac4b6d9b0c30fa4cc8ca2a-e8cb37039be049cf9c4e53bbcfbd80d9 received from {username='slave'} at 127.0.99.131:54483
I20240503 02:00:01.866333 9777 tablet_copy_service.cc:433] P db1a0000aaa346f79c0aa4b6c5105191: ending tablet copy session c688f3df56ac4b6d9b0c30fa4cc8ca2a-e8cb37039be049cf9c4e53bbcfbd80d9 on tablet e8cb37039be049cf9c4e53bbcfbd80d9 with peer c688f3df56ac4b6d9b0c30fa4cc8ca2a
I20240503 02:00:01.867313 7034 tablet_copy_service.cc:140] P 63331f455be049c1bdd6c23c59ac1bef: Received BeginTabletCopySession request for tablet aaeebd4c884445ba9f44ac9bfcb9c7ba from peer 1d0e738c34af40fa9950c6f086619b3a ({username='slave'} at 127.0.99.132:48463)
I20240503 02:00:01.867434 7034 tablet_copy_service.cc:161] P 63331f455be049c1bdd6c23c59ac1bef: Beginning new tablet copy session on tablet aaeebd4c884445ba9f44ac9bfcb9c7ba from peer 1d0e738c34af40fa9950c6f086619b3a at {username='slave'} at 127.0.99.132:48463: session id = 1d0e738c34af40fa9950c6f086619b3a-aaeebd4c884445ba9f44ac9bfcb9c7ba
I20240503 02:00:01.868965 7034 tablet_copy_source_session.cc:205] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:01.871557 17558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaeebd4c884445ba9f44ac9bfcb9c7ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.873759 17558 tablet_copy_client.cc:783] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:01.876391 17687 tablet_bootstrap.cc:492] T 3e824fcc9aac449d9373655b304010e2 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:01.876717 17687 tablet_bootstrap.cc:492] T 3e824fcc9aac449d9373655b304010e2 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap complete.
I20240503 02:00:01.876904 17687 ts_tablet_manager.cc:1367] T 3e824fcc9aac449d9373655b304010e2 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.000s
I20240503 02:00:01.877434 17687 raft_consensus.cc:348] T 3e824fcc9aac449d9373655b304010e2 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:01.877544 17687 raft_consensus.cc:729] T 3e824fcc9aac449d9373655b304010e2 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.877822 17558 tablet_copy_client.cc:647] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:01.877951 17687 consensus_queue.cc:260] T 3e824fcc9aac449d9373655b304010e2 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:01.881155 17687 ts_tablet_manager.cc:1398] T 3e824fcc9aac449d9373655b304010e2 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:00:01.881892 9514 tablet_copy_service.cc:342] P bb5364d730cb4489a0bfc6c270b7450a: Request end of tablet copy session 63331f455be049c1bdd6c23c59ac1bef-3e824fcc9aac449d9373655b304010e2 received from {username='slave'} at 127.0.99.130:54311
I20240503 02:00:01.881979 9514 tablet_copy_service.cc:433] P bb5364d730cb4489a0bfc6c270b7450a: ending tablet copy session 63331f455be049c1bdd6c23c59ac1bef-3e824fcc9aac449d9373655b304010e2 on tablet 3e824fcc9aac449d9373655b304010e2 with peer 63331f455be049c1bdd6c23c59ac1bef
I20240503 02:00:01.892180 6737 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:01.892452 6737 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:01.894587 6737 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 4da3d94fdd1c42b2bc9cb08216bb84dc
I20240503 02:00:01.895244 9342 tablet_service.cc:1514] Processing DeleteTablet for tablet 4da3d94fdd1c42b2bc9cb08216bb84dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:01 UTC) from {username='slave'} at 127.0.0.1:36040
I20240503 02:00:01.895702 9472 tablet_service.cc:1514] Processing DeleteTablet for tablet 4da3d94fdd1c42b2bc9cb08216bb84dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:01 UTC) from {username='slave'} at 127.0.0.1:54678
I20240503 02:00:01.896119 6863 tablet_service.cc:1514] Processing DeleteTablet for tablet 4da3d94fdd1c42b2bc9cb08216bb84dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:01 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 02:00:01.896384 17708 tablet_replica.cc:330] T 4da3d94fdd1c42b2bc9cb08216bb84dc P bb5364d730cb4489a0bfc6c270b7450a: stopping tablet replica
I20240503 02:00:01.896535 17708 raft_consensus.cc:2229] T 4da3d94fdd1c42b2bc9cb08216bb84dc P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.896667 17708 raft_consensus.cc:2258] T 4da3d94fdd1c42b2bc9cb08216bb84dc P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.897334 17708 ts_tablet_manager.cc:1875] T 4da3d94fdd1c42b2bc9cb08216bb84dc P bb5364d730cb4489a0bfc6c270b7450a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:01.901283 17708 ts_tablet_manager.cc:1888] T 4da3d94fdd1c42b2bc9cb08216bb84dc P bb5364d730cb4489a0bfc6c270b7450a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:01.901293 17558 tablet_copy_client.cc:523] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:01.901342 17707 tablet_replica.cc:330] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8: stopping tablet replica
I20240503 02:00:01.901505 17708 log.cc:1196] T 4da3d94fdd1c42b2bc9cb08216bb84dc P bb5364d730cb4489a0bfc6c270b7450a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/4da3d94fdd1c42b2bc9cb08216bb84dc
I20240503 02:00:01.901494 17707 raft_consensus.cc:2229] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:01.901721 17707 raft_consensus.cc:2258] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.901973 17708 ts_tablet_manager.cc:1909] T 4da3d94fdd1c42b2bc9cb08216bb84dc P bb5364d730cb4489a0bfc6c270b7450a: Deleting consensus metadata
I20240503 02:00:01.902004 17709 tablet_replica.cc:330] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 02:00:01.902205 17709 raft_consensus.cc:2229] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:01.902447 17709 raft_consensus.cc:2258] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:01.902736 17558 tablet_bootstrap.cc:492] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 02:00:01.903458 17709 ts_tablet_manager.cc:1875] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:01.905783 17707 ts_tablet_manager.cc:1875] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:01.905761 6725 catalog_manager.cc:4830] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): tablet 4da3d94fdd1c42b2bc9cb08216bb84dc (table rebalancer_dynamic_table_new_name [id=3bbac19b98664efeaf8f0aafd4e7655f]) successfully deleted
I20240503 02:00:01.907449 17709 ts_tablet_manager.cc:1888] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:01.907769 17709 log.cc:1196] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/4da3d94fdd1c42b2bc9cb08216bb84dc
I20240503 02:00:01.908295 17709 ts_tablet_manager.cc:1909] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 02:00:01.908797 17707 ts_tablet_manager.cc:1888] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:01.909314 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 4da3d94fdd1c42b2bc9cb08216bb84dc (table rebalancer_dynamic_table_new_name [id=3bbac19b98664efeaf8f0aafd4e7655f]) successfully deleted
I20240503 02:00:01.910912 17707 log.cc:1196] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/4da3d94fdd1c42b2bc9cb08216bb84dc
I20240503 02:00:01.911310 17707 ts_tablet_manager.cc:1909] T 4da3d94fdd1c42b2bc9cb08216bb84dc P 12e8c683db054df3b75c6285b98a5ad8: Deleting consensus metadata
I20240503 02:00:01.912107 6727 catalog_manager.cc:4830] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): tablet 4da3d94fdd1c42b2bc9cb08216bb84dc (table rebalancer_dynamic_table_new_name [id=3bbac19b98664efeaf8f0aafd4e7655f]) successfully deleted
I20240503 02:00:01.920178 17558 tablet_bootstrap.cc:492] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:01.920466 17558 tablet_bootstrap.cc:492] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap complete.
I20240503 02:00:01.920631 17558 ts_tablet_manager.cc:1367] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240503 02:00:01.921119 17558 raft_consensus.cc:348] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:01.921242 17558 raft_consensus.cc:729] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.921370 17558 consensus_queue.cc:260] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:01.921643 17558 ts_tablet_manager.cc:1398] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:01.922696 7034 tablet_copy_service.cc:342] P 63331f455be049c1bdd6c23c59ac1bef: Request end of tablet copy session 1d0e738c34af40fa9950c6f086619b3a-aaeebd4c884445ba9f44ac9bfcb9c7ba received from {username='slave'} at 127.0.99.132:48463
I20240503 02:00:01.922788 7034 tablet_copy_service.cc:433] P 63331f455be049c1bdd6c23c59ac1bef: ending tablet copy session 1d0e738c34af40fa9950c6f086619b3a-aaeebd4c884445ba9f44ac9bfcb9c7ba on tablet aaeebd4c884445ba9f44ac9bfcb9c7ba with peer 1d0e738c34af40fa9950c6f086619b3a
I20240503 02:00:01.959669 17415 ts_tablet_manager.cc:902] T 5e20b869a89e44f1b0ba854163c31f06 P 5dcc33cc621a4cbbae155f3230fb7c46: Initiating tablet copy from peer 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 02:00:01.960219 17415 tablet_copy_client.cc:308] T 5e20b869a89e44f1b0ba854163c31f06 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:37231
I20240503 02:00:01.960760 7034 tablet_copy_service.cc:140] P 63331f455be049c1bdd6c23c59ac1bef: Received BeginTabletCopySession request for tablet 5e20b869a89e44f1b0ba854163c31f06 from peer 5dcc33cc621a4cbbae155f3230fb7c46 ({username='slave'} at 127.0.99.129:33603)
I20240503 02:00:01.960860 7034 tablet_copy_service.cc:161] P 63331f455be049c1bdd6c23c59ac1bef: Beginning new tablet copy session on tablet 5e20b869a89e44f1b0ba854163c31f06 from peer 5dcc33cc621a4cbbae155f3230fb7c46 at {username='slave'} at 127.0.99.129:33603: session id = 5dcc33cc621a4cbbae155f3230fb7c46-5e20b869a89e44f1b0ba854163c31f06
I20240503 02:00:01.961850 7034 tablet_copy_source_session.cc:205] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:01.962747 17415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e20b869a89e44f1b0ba854163c31f06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:01.964896 17415 tablet_copy_client.cc:783] T 5e20b869a89e44f1b0ba854163c31f06 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:01.965600 17415 tablet_copy_client.cc:647] T 5e20b869a89e44f1b0ba854163c31f06 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:01.967083 17415 tablet_copy_client.cc:523] T 5e20b869a89e44f1b0ba854163c31f06 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:01.968647 17415 tablet_bootstrap.cc:492] T 5e20b869a89e44f1b0ba854163c31f06 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 02:00:01.973400 17415 tablet_bootstrap.cc:492] T 5e20b869a89e44f1b0ba854163c31f06 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:01.973676 17415 tablet_bootstrap.cc:492] T 5e20b869a89e44f1b0ba854163c31f06 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap complete.
I20240503 02:00:01.973847 17415 ts_tablet_manager.cc:1367] T 5e20b869a89e44f1b0ba854163c31f06 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:00:01.974395 17415 raft_consensus.cc:348] T 5e20b869a89e44f1b0ba854163c31f06 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:01.974571 17415 raft_consensus.cc:729] T 5e20b869a89e44f1b0ba854163c31f06 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 02:00:01.974812 17415 consensus_queue.cc:260] T 5e20b869a89e44f1b0ba854163c31f06 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:01.975214 17415 ts_tablet_manager.cc:1398] T 5e20b869a89e44f1b0ba854163c31f06 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 02:00:01.975800 7034 tablet_copy_service.cc:342] P 63331f455be049c1bdd6c23c59ac1bef: Request end of tablet copy session 5dcc33cc621a4cbbae155f3230fb7c46-5e20b869a89e44f1b0ba854163c31f06 received from {username='slave'} at 127.0.99.129:33603
I20240503 02:00:01.975883 7034 tablet_copy_service.cc:433] P 63331f455be049c1bdd6c23c59ac1bef: ending tablet copy session 5dcc33cc621a4cbbae155f3230fb7c46-5e20b869a89e44f1b0ba854163c31f06 on tablet 5e20b869a89e44f1b0ba854163c31f06 with peer 5dcc33cc621a4cbbae155f3230fb7c46
I20240503 02:00:01.992372 9625 raft_consensus.cc:1203] T 22b6ec13c4e04d6cbade150f0c644e52 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:01.999986 6732 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:02.012403 9736 tablet_service.cc:1467] Processing CreateTablet for tablet 74a2800b046f4b2abef0cfb7c0833a7a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2f5881aee73144a6acfd802d3fd8b1e9]), partition=
I20240503 02:00:02.012509 6863 tablet_service.cc:1467] Processing CreateTablet for tablet 74a2800b046f4b2abef0cfb7c0833a7a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2f5881aee73144a6acfd802d3fd8b1e9]), partition=
I20240503 02:00:02.012822 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74a2800b046f4b2abef0cfb7c0833a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.012840 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74a2800b046f4b2abef0cfb7c0833a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.015398 17495 tablet_bootstrap.cc:492] T 74a2800b046f4b2abef0cfb7c0833a7a P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 02:00:02.016171 17752 tablet_bootstrap.cc:492] T 74a2800b046f4b2abef0cfb7c0833a7a P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 02:00:02.016458 17495 tablet_bootstrap.cc:654] T 74a2800b046f4b2abef0cfb7c0833a7a P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.017108 17752 tablet_bootstrap.cc:654] T 74a2800b046f4b2abef0cfb7c0833a7a P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.018276 9757 raft_consensus.cc:1203] T 76e8697ac8624e03a46bddae35f4c9fa P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 02:00:02.018358 7124 tablet_service.cc:1467] Processing CreateTablet for tablet 74a2800b046f4b2abef0cfb7c0833a7a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2f5881aee73144a6acfd802d3fd8b1e9]), partition=
I20240503 02:00:02.018687 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74a2800b046f4b2abef0cfb7c0833a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.020888 17752 tablet_bootstrap.cc:492] T 74a2800b046f4b2abef0cfb7c0833a7a P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 02:00:02.021034 17752 ts_tablet_manager.cc:1367] T 74a2800b046f4b2abef0cfb7c0833a7a P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:02.021827 9361 raft_consensus.cc:1203] T e811662836164c3fb29a1b12ace7eb41 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 02:00:02.022065 17495 tablet_bootstrap.cc:492] T 74a2800b046f4b2abef0cfb7c0833a7a P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 02:00:02.022147 17495 ts_tablet_manager.cc:1367] T 74a2800b046f4b2abef0cfb7c0833a7a P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:02.022655 17495 raft_consensus.cc:348] T 74a2800b046f4b2abef0cfb7c0833a7a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:02.022789 17495 raft_consensus.cc:374] T 74a2800b046f4b2abef0cfb7c0833a7a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:02.022830 17495 raft_consensus.cc:729] T 74a2800b046f4b2abef0cfb7c0833a7a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.022940 17495 consensus_queue.cc:260] T 74a2800b046f4b2abef0cfb7c0833a7a P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:02.023218 17495 ts_tablet_manager.cc:1398] T 74a2800b046f4b2abef0cfb7c0833a7a P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:02.024314 17752 raft_consensus.cc:348] T 74a2800b046f4b2abef0cfb7c0833a7a P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:02.024881 17752 raft_consensus.cc:374] T 74a2800b046f4b2abef0cfb7c0833a7a P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:02.024961 17752 raft_consensus.cc:729] T 74a2800b046f4b2abef0cfb7c0833a7a P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.025142 17752 consensus_queue.cc:260] T 74a2800b046f4b2abef0cfb7c0833a7a P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:02.030390 17692 ts_tablet_manager.cc:902] T 7046186cd7f540e7b0292da5f422a4a1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Initiating tablet copy from peer 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 02:00:02.031275 17692 tablet_copy_client.cc:308] T 7046186cd7f540e7b0292da5f422a4a1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36271
I20240503 02:00:02.032543 17755 tablet_bootstrap.cc:492] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 02:00:02.032892 7295 tablet_copy_service.cc:140] P 1d0e738c34af40fa9950c6f086619b3a: Received BeginTabletCopySession request for tablet 7046186cd7f540e7b0292da5f422a4a1 from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a ({username='slave'} at 127.0.99.131:46745)
I20240503 02:00:02.033018 7295 tablet_copy_service.cc:161] P 1d0e738c34af40fa9950c6f086619b3a: Beginning new tablet copy session on tablet 7046186cd7f540e7b0292da5f422a4a1 from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a at {username='slave'} at 127.0.99.131:46745: session id = c688f3df56ac4b6d9b0c30fa4cc8ca2a-7046186cd7f540e7b0292da5f422a4a1
I20240503 02:00:02.034112 7295 tablet_copy_source_session.cc:205] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:02.037164 17692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7046186cd7f540e7b0292da5f422a4a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.039988 17755 tablet_bootstrap.cc:654] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.025535 17752 ts_tablet_manager.cc:1398] T 74a2800b046f4b2abef0cfb7c0833a7a P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:00:02.040040 17692 tablet_copy_client.cc:783] T 7046186cd7f540e7b0292da5f422a4a1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:02.042026 17692 tablet_copy_client.cc:647] T 7046186cd7f540e7b0292da5f422a4a1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:02.048895 17755 tablet_bootstrap.cc:492] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 02:00:02.049085 17755 ts_tablet_manager.cc:1367] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240503 02:00:02.049643 17755 raft_consensus.cc:348] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:02.049753 17755 raft_consensus.cc:374] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:02.049788 17755 raft_consensus.cc:729] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.049928 17755 consensus_queue.cc:260] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:02.050243 17755 ts_tablet_manager.cc:1398] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:02.051673 17692 tablet_copy_client.cc:523] T 7046186cd7f540e7b0292da5f422a4a1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:02.053339 17692 tablet_bootstrap.cc:492] T 7046186cd7f540e7b0292da5f422a4a1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 02:00:02.058439 17692 tablet_bootstrap.cc:492] T 7046186cd7f540e7b0292da5f422a4a1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:02.058774 17692 tablet_bootstrap.cc:492] T 7046186cd7f540e7b0292da5f422a4a1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap complete.
I20240503 02:00:02.061246 17692 ts_tablet_manager.cc:1367] T 7046186cd7f540e7b0292da5f422a4a1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240503 02:00:02.062222 17692 raft_consensus.cc:348] T 7046186cd7f540e7b0292da5f422a4a1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:02.062332 17692 raft_consensus.cc:729] T 7046186cd7f540e7b0292da5f422a4a1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.062472 17692 consensus_queue.cc:260] T 7046186cd7f540e7b0292da5f422a4a1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:02.062881 17692 ts_tablet_manager.cc:1398] T 7046186cd7f540e7b0292da5f422a4a1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:02.063879 7295 tablet_copy_service.cc:342] P 1d0e738c34af40fa9950c6f086619b3a: Request end of tablet copy session c688f3df56ac4b6d9b0c30fa4cc8ca2a-7046186cd7f540e7b0292da5f422a4a1 received from {username='slave'} at 127.0.99.131:46745
I20240503 02:00:02.064036 7295 tablet_copy_service.cc:433] P 1d0e738c34af40fa9950c6f086619b3a: ending tablet copy session c688f3df56ac4b6d9b0c30fa4cc8ca2a-7046186cd7f540e7b0292da5f422a4a1 on tablet 7046186cd7f540e7b0292da5f422a4a1 with peer c688f3df56ac4b6d9b0c30fa4cc8ca2a
I20240503 02:00:02.072804 9493 raft_consensus.cc:1203] T d80398c9a264430197262bb42241d58d P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:02.092065 15159 raft_consensus.cc:482] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:02.092233 15159 raft_consensus.cc:504] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:02.092757 15159 leader_election.cc:288] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 02:00:02.093297 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a2800b046f4b2abef0cfb7c0833a7a" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 02:00:02.093449 6882 raft_consensus.cc:2454] T 74a2800b046f4b2abef0cfb7c0833a7a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 02:00:02.093827 7081 leader_election.cc:302] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 02:00:02.094307 15159 raft_consensus.cc:2781] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:02.094360 15159 raft_consensus.cc:482] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:02.094396 15159 raft_consensus.cc:3037] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:02.095252 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a2800b046f4b2abef0cfb7c0833a7a" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 02:00:02.095456 9757 raft_consensus.cc:2454] T 74a2800b046f4b2abef0cfb7c0833a7a P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 02:00:02.096249 15159 raft_consensus.cc:504] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:02.096668 15159 leader_election.cc:288] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Requested vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 02:00:02.097326 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a2800b046f4b2abef0cfb7c0833a7a" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 02:00:02.097489 6882 raft_consensus.cc:3037] T 74a2800b046f4b2abef0cfb7c0833a7a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:02.099563 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a2800b046f4b2abef0cfb7c0833a7a" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 02:00:02.099725 9757 raft_consensus.cc:3037] T 74a2800b046f4b2abef0cfb7c0833a7a P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:02.101318 9757 raft_consensus.cc:2454] T 74a2800b046f4b2abef0cfb7c0833a7a P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 02:00:02.102203 7079 leader_election.cc:302] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 02:00:02.102691 16780 raft_consensus.cc:2781] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:02.102772 16780 raft_consensus.cc:686] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Becoming Leader. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Running, Role: LEADER
I20240503 02:00:02.102912 16780 consensus_queue.cc:237] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:02.107573 6882 raft_consensus.cc:2454] T 74a2800b046f4b2abef0cfb7c0833a7a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 02:00:02.109846 6737 catalog_manager.cc:5484] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:02.130398 9342 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:00:02.142947 6992 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:00:02.143538 7254 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:00:02.154275 9472 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:02.160912 9361 raft_consensus.cc:1203] T fd29f3b059874b019153b64ef8aaaa48 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:02.184965 6863 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:02.186332 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:02.200176 7124 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:00:02.218082 17557 consensus_queue.cc:1035] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:02.222465 6737 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:02.230435 7145 raft_consensus.cc:1203] T e8cb37039be049cf9c4e53bbcfbd80d9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:02.232542 9736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:00:02.234983 6882 raft_consensus.cc:1261] T 74a2800b046f4b2abef0cfb7c0833a7a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:02.235178 17364 consensus_queue.cc:1035] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:02.235687 9757 raft_consensus.cc:1261] T 74a2800b046f4b2abef0cfb7c0833a7a P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:02.236202 15159 consensus_queue.cc:1035] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:02.236511 15159 consensus_queue.cc:1035] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:02.241199 6737 catalog_manager.cc:5576] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943) does not have the latest schema for tablet 74a2800b046f4b2abef0cfb7c0833a7a (table rebalancer_dynamic_table [id=2f5881aee73144a6acfd802d3fd8b1e9]). Expected version 1 got 0
I20240503 02:00:02.242130 6741 catalog_manager.cc:5576] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053) does not have the latest schema for tablet 74a2800b046f4b2abef0cfb7c0833a7a (table rebalancer_dynamic_table [id=2f5881aee73144a6acfd802d3fd8b1e9]). Expected version 1 got 0
I20240503 02:00:02.246734 17616 tablet.cc:1722] T 74a2800b046f4b2abef0cfb7c0833a7a P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:02.247110 17616 tablet.cc:1620] T 74a2800b046f4b2abef0cfb7c0833a7a P 5dcc33cc621a4cbbae155f3230fb7c46: MemRowSet was empty: no flush needed.
I20240503 02:00:02.250699 17784 tablet.cc:1722] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:02.251411 17784 tablet.cc:1620] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: MemRowSet was empty: no flush needed.
I20240503 02:00:02.260522 6736 catalog_manager.cc:5576] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053) does not have the latest schema for tablet 74a2800b046f4b2abef0cfb7c0833a7a (table rebalancer_dynamic_table [id=2f5881aee73144a6acfd802d3fd8b1e9]). Expected version 1 got 0
I20240503 02:00:02.264123 17785 tablet.cc:1722] T 74a2800b046f4b2abef0cfb7c0833a7a P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:02.264628 17785 tablet.cc:1620] T 74a2800b046f4b2abef0cfb7c0833a7a P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 02:00:02.266659 6737 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=2f5881aee73144a6acfd802d3fd8b1e9] alter complete (version 1)
I20240503 02:00:02.283170 7274 raft_consensus.cc:1203] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:02.287755 7014 raft_consensus.cc:1203] T 3e824fcc9aac449d9373655b304010e2 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:00:02.303258 17147 consensus_queue.cc:1035] T 9cc9a609452f4f4aad600381968b9227 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:02.309168 17147 consensus_queue.cc:1035] T 9cc9a609452f4f4aad600381968b9227 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:02.365823 6740 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:02.371686 6736 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:02.375183 9472 tablet_service.cc:1467] Processing CreateTablet for tablet d91783037d994d8882ae6e835b70ae39 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=3021455fb6dc4f84a1899807090aab26]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:02.375624 9472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d91783037d994d8882ae6e835b70ae39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.376168 7124 tablet_service.cc:1467] Processing CreateTablet for tablet 06bffbde7b9f4b31835f61795e6ca32a (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=3021455fb6dc4f84a1899807090aab26]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:02.376168 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 5097893b51da40059b0ef865d42c4d4a (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=3021455fb6dc4f84a1899807090aab26]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:02.376358 6863 tablet_service.cc:1467] Processing CreateTablet for tablet 5097893b51da40059b0ef865d42c4d4a (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=3021455fb6dc4f84a1899807090aab26]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:02.376436 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06bffbde7b9f4b31835f61795e6ca32a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.376478 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5097893b51da40059b0ef865d42c4d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.376603 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5097893b51da40059b0ef865d42c4d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.379272 6992 tablet_service.cc:1467] Processing CreateTablet for tablet d91783037d994d8882ae6e835b70ae39 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=3021455fb6dc4f84a1899807090aab26]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:02.379583 6992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d91783037d994d8882ae6e835b70ae39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.379606 17755 tablet_bootstrap.cc:492] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 02:00:02.379602 17495 tablet_bootstrap.cc:492] T 5097893b51da40059b0ef865d42c4d4a P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 02:00:02.380578 17495 tablet_bootstrap.cc:654] T 5097893b51da40059b0ef865d42c4d4a P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.380560 17755 tablet_bootstrap.cc:654] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.380802 17831 tablet_bootstrap.cc:492] T d91783037d994d8882ae6e835b70ae39 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 02:00:02.385356 17831 tablet_bootstrap.cc:654] T d91783037d994d8882ae6e835b70ae39 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.387666 9472 tablet_service.cc:1467] Processing CreateTablet for tablet 06bffbde7b9f4b31835f61795e6ca32a (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=3021455fb6dc4f84a1899807090aab26]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:02.388022 9472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06bffbde7b9f4b31835f61795e6ca32a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.389225 16102 consensus_queue.cc:1035] T c9173280a825443db6088e7141217b29 P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:02.392239 17835 tablet_bootstrap.cc:492] T 5097893b51da40059b0ef865d42c4d4a P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 02:00:02.394104 17495 tablet_bootstrap.cc:492] T 5097893b51da40059b0ef865d42c4d4a P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 02:00:02.394232 17495 ts_tablet_manager.cc:1367] T 5097893b51da40059b0ef865d42c4d4a P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 02:00:02.394910 17834 tablet_bootstrap.cc:492] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 02:00:02.395027 17495 raft_consensus.cc:348] T 5097893b51da40059b0ef865d42c4d4a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:02.395154 17495 raft_consensus.cc:374] T 5097893b51da40059b0ef865d42c4d4a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:02.395190 17495 raft_consensus.cc:729] T 5097893b51da40059b0ef865d42c4d4a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.395407 17784 tablet.cc:1722] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:02.395764 17784 tablet.cc:1620] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: MemRowSet was empty: no flush needed.
I20240503 02:00:02.395740 17616 tablet.cc:1722] T 74a2800b046f4b2abef0cfb7c0833a7a P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:02.396039 17616 tablet.cc:1620] T 74a2800b046f4b2abef0cfb7c0833a7a P 5dcc33cc621a4cbbae155f3230fb7c46: MemRowSet was empty: no flush needed.
I20240503 02:00:02.396294 17495 consensus_queue.cc:260] T 5097893b51da40059b0ef865d42c4d4a P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:02.396584 9736 tablet_service.cc:1467] Processing CreateTablet for tablet 5097893b51da40059b0ef865d42c4d4a (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=3021455fb6dc4f84a1899807090aab26]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:02.396890 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5097893b51da40059b0ef865d42c4d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.397609 7255 tablet_service.cc:1467] Processing CreateTablet for tablet 06bffbde7b9f4b31835f61795e6ca32a (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=3021455fb6dc4f84a1899807090aab26]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:02.397768 17495 ts_tablet_manager.cc:1398] T 5097893b51da40059b0ef865d42c4d4a P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:02.397888 7255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06bffbde7b9f4b31835f61795e6ca32a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.399513 7254 tablet_service.cc:1467] Processing CreateTablet for tablet d91783037d994d8882ae6e835b70ae39 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=3021455fb6dc4f84a1899807090aab26]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:02.399760 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d91783037d994d8882ae6e835b70ae39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.401154 17755 tablet_bootstrap.cc:492] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 02:00:02.401242 17755 ts_tablet_manager.cc:1367] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240503 02:00:02.401834 17752 tablet_bootstrap.cc:492] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 02:00:02.401746 17755 raft_consensus.cc:348] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:02.401880 17755 raft_consensus.cc:374] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:02.401909 17755 raft_consensus.cc:729] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.402024 17755 consensus_queue.cc:260] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:02.402330 17755 ts_tablet_manager.cc:1398] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:02.402877 17752 tablet_bootstrap.cc:654] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.403532 17834 tablet_bootstrap.cc:654] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.403602 17785 tablet.cc:1722] T 74a2800b046f4b2abef0cfb7c0833a7a P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:02.403877 17785 tablet.cc:1620] T 74a2800b046f4b2abef0cfb7c0833a7a P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 02:00:02.405133 17835 tablet_bootstrap.cc:654] T 5097893b51da40059b0ef865d42c4d4a P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.406188 16102 consensus_queue.cc:1035] T c9173280a825443db6088e7141217b29 P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:02.412581 6882 raft_consensus.cc:1203] T 5e20b869a89e44f1b0ba854163c31f06 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:02.413126 17835 tablet_bootstrap.cc:492] T 5097893b51da40059b0ef865d42c4d4a P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 02:00:02.413314 17835 ts_tablet_manager.cc:1367] T 5097893b51da40059b0ef865d42c4d4a P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20240503 02:00:02.415041 17835 raft_consensus.cc:348] T 5097893b51da40059b0ef865d42c4d4a P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:02.415117 6736 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=2f5881aee73144a6acfd802d3fd8b1e9] alter complete (version 2)
I20240503 02:00:02.415269 17835 raft_consensus.cc:374] T 5097893b51da40059b0ef865d42c4d4a P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:02.415359 17835 raft_consensus.cc:729] T 5097893b51da40059b0ef865d42c4d4a P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.415474 17837 tablet_bootstrap.cc:492] T 06bffbde7b9f4b31835f61795e6ca32a P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 02:00:02.416968 17752 tablet_bootstrap.cc:492] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 02:00:02.417084 17752 ts_tablet_manager.cc:1367] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240503 02:00:02.417584 17752 raft_consensus.cc:348] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:02.417696 17752 raft_consensus.cc:374] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:02.417726 17752 raft_consensus.cc:729] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.417847 17752 consensus_queue.cc:260] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:02.418131 17752 ts_tablet_manager.cc:1398] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:02.419279 17837 tablet_bootstrap.cc:654] T 06bffbde7b9f4b31835f61795e6ca32a P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.420746 17835 consensus_queue.cc:260] T 5097893b51da40059b0ef865d42c4d4a P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:02.421531 17835 ts_tablet_manager.cc:1398] T 5097893b51da40059b0ef865d42c4d4a P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240503 02:00:02.422339 17831 tablet_bootstrap.cc:492] T d91783037d994d8882ae6e835b70ae39 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 02:00:02.422436 17831 ts_tablet_manager.cc:1367] T d91783037d994d8882ae6e835b70ae39 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.000s
I20240503 02:00:02.422899 17834 tablet_bootstrap.cc:492] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 02:00:02.422919 17831 raft_consensus.cc:348] T d91783037d994d8882ae6e835b70ae39 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:02.423022 17831 raft_consensus.cc:374] T d91783037d994d8882ae6e835b70ae39 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:02.423040 17834 ts_tablet_manager.cc:1367] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.000s
I20240503 02:00:02.423049 17831 raft_consensus.cc:729] T d91783037d994d8882ae6e835b70ae39 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.423179 17831 consensus_queue.cc:260] T d91783037d994d8882ae6e835b70ae39 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:02.423427 17831 ts_tablet_manager.cc:1398] T d91783037d994d8882ae6e835b70ae39 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:02.423715 17831 tablet_bootstrap.cc:492] T 06bffbde7b9f4b31835f61795e6ca32a P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 02:00:02.424494 17834 raft_consensus.cc:348] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:02.424636 17834 raft_consensus.cc:374] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:02.424669 17834 raft_consensus.cc:729] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.424731 17831 tablet_bootstrap.cc:654] T 06bffbde7b9f4b31835f61795e6ca32a P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.424998 17834 consensus_queue.cc:260] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:02.432211 17837 tablet_bootstrap.cc:492] T 06bffbde7b9f4b31835f61795e6ca32a P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 02:00:02.432408 17837 ts_tablet_manager.cc:1367] T 06bffbde7b9f4b31835f61795e6ca32a P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240503 02:00:02.433005 17837 raft_consensus.cc:348] T 06bffbde7b9f4b31835f61795e6ca32a P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:02.433188 17837 raft_consensus.cc:374] T 06bffbde7b9f4b31835f61795e6ca32a P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:02.433261 17837 raft_consensus.cc:729] T 06bffbde7b9f4b31835f61795e6ca32a P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.433418 17837 consensus_queue.cc:260] T 06bffbde7b9f4b31835f61795e6ca32a P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:02.433785 17837 ts_tablet_manager.cc:1398] T 06bffbde7b9f4b31835f61795e6ca32a P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:02.434062 17837 tablet_bootstrap.cc:492] T d91783037d994d8882ae6e835b70ae39 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 02:00:02.435829 17834 ts_tablet_manager.cc:1398] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240503 02:00:02.440754 17837 tablet_bootstrap.cc:654] T d91783037d994d8882ae6e835b70ae39 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.445644 17837 tablet_bootstrap.cc:492] T d91783037d994d8882ae6e835b70ae39 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 02:00:02.445775 17837 ts_tablet_manager.cc:1367] T d91783037d994d8882ae6e835b70ae39 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:00:02.447510 17837 raft_consensus.cc:348] T d91783037d994d8882ae6e835b70ae39 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:02.447759 17837 raft_consensus.cc:374] T d91783037d994d8882ae6e835b70ae39 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:02.447862 17837 raft_consensus.cc:729] T d91783037d994d8882ae6e835b70ae39 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.448097 17837 consensus_queue.cc:260] T d91783037d994d8882ae6e835b70ae39 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:02.451175 17511 raft_consensus.cc:482] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:02.451308 17511 raft_consensus.cc:504] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:02.451781 17511 leader_election.cc:288] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 02:00:02.452391 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d91783037d994d8882ae6e835b70ae39" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 02:00:02.452524 7274 raft_consensus.cc:2454] T d91783037d994d8882ae6e835b70ae39 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 02:00:02.452831 6948 leader_election.cc:302] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 02:00:02.452898 17837 ts_tablet_manager.cc:1398] T d91783037d994d8882ae6e835b70ae39 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:02.453058 7145 raft_consensus.cc:1203] T 7046186cd7f540e7b0292da5f422a4a1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:02.453198 17508 raft_consensus.cc:2781] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:02.453250 17508 raft_consensus.cc:482] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:02.453289 17508 raft_consensus.cc:3037] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:02.456705 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d91783037d994d8882ae6e835b70ae39" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 02:00:02.456877 9493 raft_consensus.cc:2454] T d91783037d994d8882ae6e835b70ae39 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 02:00:02.457531 17831 tablet_bootstrap.cc:492] T 06bffbde7b9f4b31835f61795e6ca32a P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 02:00:02.457677 17831 ts_tablet_manager.cc:1367] T 06bffbde7b9f4b31835f61795e6ca32a P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20240503 02:00:02.457731 17508 raft_consensus.cc:504] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:02.458130 17508 leader_election.cc:288] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 02:00:02.458192 17831 raft_consensus.cc:348] T 06bffbde7b9f4b31835f61795e6ca32a P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:02.458302 17831 raft_consensus.cc:374] T 06bffbde7b9f4b31835f61795e6ca32a P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:02.458334 17831 raft_consensus.cc:729] T 06bffbde7b9f4b31835f61795e6ca32a P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.458447 17831 consensus_queue.cc:260] T 06bffbde7b9f4b31835f61795e6ca32a P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:02.458782 17831 ts_tablet_manager.cc:1398] T 06bffbde7b9f4b31835f61795e6ca32a P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:02.460186 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d91783037d994d8882ae6e835b70ae39" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 02:00:02.460340 7274 raft_consensus.cc:3037] T d91783037d994d8882ae6e835b70ae39 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:02.461584 7274 raft_consensus.cc:2454] T d91783037d994d8882ae6e835b70ae39 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 02:00:02.462167 6948 leader_election.cc:302] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 02:00:02.462366 17508 raft_consensus.cc:2781] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:02.462452 17508 raft_consensus.cc:686] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 02:00:02.462618 17508 consensus_queue.cc:237] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:02.463089 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d91783037d994d8882ae6e835b70ae39" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 02:00:02.463204 9493 raft_consensus.cc:3037] T d91783037d994d8882ae6e835b70ae39 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:02.464223 9493 raft_consensus.cc:2454] T d91783037d994d8882ae6e835b70ae39 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 02:00:02.467458 6734 catalog_manager.cc:5484] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:02.488746 17147 raft_consensus.cc:482] T 5097893b51da40059b0ef865d42c4d4a P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:02.488875 17147 raft_consensus.cc:504] T 5097893b51da40059b0ef865d42c4d4a P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:02.489245 17147 leader_election.cc:288] T 5097893b51da40059b0ef865d42c4d4a P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 02:00:02.489682 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5097893b51da40059b0ef865d42c4d4a" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 02:00:02.489722 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5097893b51da40059b0ef865d42c4d4a" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 02:00:02.489833 6882 raft_consensus.cc:2454] T 5097893b51da40059b0ef865d42c4d4a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 02:00:02.489853 9755 raft_consensus.cc:2454] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 02:00:02.490113 9297 leader_election.cc:302] T 5097893b51da40059b0ef865d42c4d4a P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 02:00:02.490444 17147 raft_consensus.cc:2781] T 5097893b51da40059b0ef865d42c4d4a P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:02.490494 17147 raft_consensus.cc:482] T 5097893b51da40059b0ef865d42c4d4a P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:02.490561 17147 raft_consensus.cc:3037] T 5097893b51da40059b0ef865d42c4d4a P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:02.491701 17147 raft_consensus.cc:504] T 5097893b51da40059b0ef865d42c4d4a P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:02.492034 17147 leader_election.cc:288] T 5097893b51da40059b0ef865d42c4d4a P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Requested vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 02:00:02.492846 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5097893b51da40059b0ef865d42c4d4a" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 02:00:02.492961 9755 raft_consensus.cc:3037] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:02.493942 9755 raft_consensus.cc:2454] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 02:00:02.494345 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5097893b51da40059b0ef865d42c4d4a" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 02:00:02.494488 6882 raft_consensus.cc:3037] T 5097893b51da40059b0ef865d42c4d4a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:02.495543 6882 raft_consensus.cc:2454] T 5097893b51da40059b0ef865d42c4d4a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 02:00:02.496042 9299 leader_election.cc:302] T 5097893b51da40059b0ef865d42c4d4a P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 02:00:02.497056 17147 raft_consensus.cc:2781] T 5097893b51da40059b0ef865d42c4d4a P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:02.497136 17147 raft_consensus.cc:686] T 5097893b51da40059b0ef865d42c4d4a P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Running, Role: LEADER
I20240503 02:00:02.497254 17147 consensus_queue.cc:237] T 5097893b51da40059b0ef865d42c4d4a P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:02.506201 6734 catalog_manager.cc:5484] T 5097893b51da40059b0ef865d42c4d4a P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:02.520316 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:02.528987 17784 tablet.cc:1722] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:02.535893 17616 tablet.cc:1722] T 74a2800b046f4b2abef0cfb7c0833a7a P 5dcc33cc621a4cbbae155f3230fb7c46: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:02.536352 17785 tablet.cc:1722] T 74a2800b046f4b2abef0cfb7c0833a7a P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:02.537812 6734 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=2f5881aee73144a6acfd802d3fd8b1e9] alter complete (version 3)
I20240503 02:00:02.545004 6863 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:02.558808 9342 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:00:02.559684 7254 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:00:02.561916 9472 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:02.563606 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:02.577106 9736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:00:02.584319 7124 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:00:02.600358 6992 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:00:02.639744 6734 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:02.639904 6734 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:02.642025 6734 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 74a2800b046f4b2abef0cfb7c0833a7a
I20240503 02:00:02.642664 9736 tablet_service.cc:1514] Processing DeleteTablet for tablet 74a2800b046f4b2abef0cfb7c0833a7a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:02 UTC) from {username='slave'} at 127.0.0.1:48152
I20240503 02:00:02.642822 7124 tablet_service.cc:1514] Processing DeleteTablet for tablet 74a2800b046f4b2abef0cfb7c0833a7a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:02 UTC) from {username='slave'} at 127.0.0.1:50370
I20240503 02:00:02.642944 6863 tablet_service.cc:1514] Processing DeleteTablet for tablet 74a2800b046f4b2abef0cfb7c0833a7a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:02 UTC) from {username='slave'} at 127.0.0.1:35310
I20240503 02:00:02.642994 17879 tablet_replica.cc:330] T 74a2800b046f4b2abef0cfb7c0833a7a P db1a0000aaa346f79c0aa4b6c5105191: stopping tablet replica
I20240503 02:00:02.643153 17879 raft_consensus.cc:2229] T 74a2800b046f4b2abef0cfb7c0833a7a P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:02.643177 17880 tablet_replica.cc:330] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 02:00:02.643241 17881 tablet_replica.cc:330] T 74a2800b046f4b2abef0cfb7c0833a7a P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 02:00:02.643307 17879 raft_consensus.cc:2258] T 74a2800b046f4b2abef0cfb7c0833a7a P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:02.643328 17880 raft_consensus.cc:2229] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:02.643613 17880 raft_consensus.cc:2258] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:02.643405 17881 raft_consensus.cc:2229] T 74a2800b046f4b2abef0cfb7c0833a7a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:02.643981 17879 ts_tablet_manager.cc:1875] T 74a2800b046f4b2abef0cfb7c0833a7a P db1a0000aaa346f79c0aa4b6c5105191: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:02.644073 17881 raft_consensus.cc:2258] T 74a2800b046f4b2abef0cfb7c0833a7a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:02.644289 17880 ts_tablet_manager.cc:1875] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:02.644767 17881 ts_tablet_manager.cc:1875] T 74a2800b046f4b2abef0cfb7c0833a7a P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:02.646883 17879 ts_tablet_manager.cc:1888] T 74a2800b046f4b2abef0cfb7c0833a7a P db1a0000aaa346f79c0aa4b6c5105191: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:02.646970 17879 log.cc:1196] T 74a2800b046f4b2abef0cfb7c0833a7a P db1a0000aaa346f79c0aa4b6c5105191: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/74a2800b046f4b2abef0cfb7c0833a7a
I20240503 02:00:02.647114 17881 ts_tablet_manager.cc:1888] T 74a2800b046f4b2abef0cfb7c0833a7a P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:02.647210 17881 log.cc:1196] T 74a2800b046f4b2abef0cfb7c0833a7a P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/74a2800b046f4b2abef0cfb7c0833a7a
I20240503 02:00:02.647311 17879 ts_tablet_manager.cc:1909] T 74a2800b046f4b2abef0cfb7c0833a7a P db1a0000aaa346f79c0aa4b6c5105191: Deleting consensus metadata
I20240503 02:00:02.647627 17881 ts_tablet_manager.cc:1909] T 74a2800b046f4b2abef0cfb7c0833a7a P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting consensus metadata
I20240503 02:00:02.648273 6726 catalog_manager.cc:4830] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): tablet 74a2800b046f4b2abef0cfb7c0833a7a (table rebalancer_dynamic_table_new_name [id=2f5881aee73144a6acfd802d3fd8b1e9]) successfully deleted
I20240503 02:00:02.648411 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 74a2800b046f4b2abef0cfb7c0833a7a (table rebalancer_dynamic_table_new_name [id=2f5881aee73144a6acfd802d3fd8b1e9]) successfully deleted
I20240503 02:00:02.652019 17880 ts_tablet_manager.cc:1888] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:02.652412 17880 log.cc:1196] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/74a2800b046f4b2abef0cfb7c0833a7a
I20240503 02:00:02.652802 17880 ts_tablet_manager.cc:1909] T 74a2800b046f4b2abef0cfb7c0833a7a P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting consensus metadata
I20240503 02:00:02.653569 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet 74a2800b046f4b2abef0cfb7c0833a7a (table rebalancer_dynamic_table_new_name [id=2f5881aee73144a6acfd802d3fd8b1e9]) successfully deleted
I20240503 02:00:02.656934 6882 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fd29f3b059874b019153b64ef8aaaa48"
dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35292
I20240503 02:00:02.657115 6882 raft_consensus.cc:595] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:00:02.659626 17203 raft_consensus.cc:979] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46: : Instructing follower 12e8c683db054df3b75c6285b98a5ad8 to start an election
I20240503 02:00:02.659752 17203 raft_consensus.cc:1067] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Signalling peer 12e8c683db054df3b75c6285b98a5ad8 to start an election
I20240503 02:00:02.660421 9361 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fd29f3b059874b019153b64ef8aaaa48"
dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
from {username='slave'} at 127.0.99.129:37661
I20240503 02:00:02.660578 9361 raft_consensus.cc:482] T fd29f3b059874b019153b64ef8aaaa48 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:02.660642 9361 raft_consensus.cc:3037] T fd29f3b059874b019153b64ef8aaaa48 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:02.661607 9361 raft_consensus.cc:504] T fd29f3b059874b019153b64ef8aaaa48 P 12e8c683db054df3b75c6285b98a5ad8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:02.662061 9361 leader_election.cc:288] T fd29f3b059874b019153b64ef8aaaa48 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 2 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 02:00:02.662462 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd29f3b059874b019153b64ef8aaaa48" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 02:00:02.662585 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd29f3b059874b019153b64ef8aaaa48" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 02:00:02.662664 7014 raft_consensus.cc:3037] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:02.662706 9625 raft_consensus.cc:3037] T fd29f3b059874b019153b64ef8aaaa48 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:02.662767 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd29f3b059874b019153b64ef8aaaa48" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 02:00:02.662880 6882 raft_consensus.cc:3032] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:02.662918 6882 raft_consensus.cc:729] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 02:00:02.663064 6882 consensus_queue.cc:260] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:02.663236 6882 raft_consensus.cc:3037] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:02.663736 9625 raft_consensus.cc:2454] T fd29f3b059874b019153b64ef8aaaa48 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 2.
I20240503 02:00:02.663770 7014 raft_consensus.cc:2454] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 2.
I20240503 02:00:02.664302 6882 raft_consensus.cc:2454] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 2.
I20240503 02:00:02.664302 9296 leader_election.cc:302] T fd29f3b059874b019153b64ef8aaaa48 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 63331f455be049c1bdd6c23c59ac1bef, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 02:00:02.664500 17147 raft_consensus.cc:2781] T fd29f3b059874b019153b64ef8aaaa48 P 12e8c683db054df3b75c6285b98a5ad8 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:02.664597 17147 raft_consensus.cc:686] T fd29f3b059874b019153b64ef8aaaa48 P 12e8c683db054df3b75c6285b98a5ad8 [term 2 LEADER]: Becoming Leader. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Running, Role: LEADER
I20240503 02:00:02.664742 17147 consensus_queue.cc:237] T fd29f3b059874b019153b64ef8aaaa48 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:02.668347 6734 catalog_manager.cc:5484] T fd29f3b059874b019153b64ef8aaaa48 P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: term changed from 1 to 2, leader changed from 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) to 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133). New cstate: current_term: 2 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:02.746073 6740 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:02.756038 9472 tablet_service.cc:1467] Processing CreateTablet for tablet 71e8030b309f456fad96592ac72da685 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=003cb51a186e417d844491ae5dee69f5]), partition=
I20240503 02:00:02.756317 9736 tablet_service.cc:1467] Processing CreateTablet for tablet 71e8030b309f456fad96592ac72da685 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=003cb51a186e417d844491ae5dee69f5]), partition=
I20240503 02:00:02.756377 9472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71e8030b309f456fad96592ac72da685. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.756449 7124 tablet_service.cc:1467] Processing CreateTablet for tablet 71e8030b309f456fad96592ac72da685 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=003cb51a186e417d844491ae5dee69f5]), partition=
I20240503 02:00:02.756549 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71e8030b309f456fad96592ac72da685. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.756691 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71e8030b309f456fad96592ac72da685. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:02.758946 17755 tablet_bootstrap.cc:492] T 71e8030b309f456fad96592ac72da685 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 02:00:02.758941 17752 tablet_bootstrap.cc:492] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 02:00:02.759724 17831 tablet_bootstrap.cc:492] T 71e8030b309f456fad96592ac72da685 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 02:00:02.759929 17755 tablet_bootstrap.cc:654] T 71e8030b309f456fad96592ac72da685 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.759929 17752 tablet_bootstrap.cc:654] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.760679 17831 tablet_bootstrap.cc:654] T 71e8030b309f456fad96592ac72da685 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:02.762549 17752 tablet_bootstrap.cc:492] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 02:00:02.762650 17752 ts_tablet_manager.cc:1367] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:02.763171 17752 raft_consensus.cc:348] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:02.763288 17752 raft_consensus.cc:374] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:02.763321 17752 raft_consensus.cc:729] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.763442 17752 consensus_queue.cc:260] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:02.763754 17752 ts_tablet_manager.cc:1398] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:02.766166 17755 tablet_bootstrap.cc:492] T 71e8030b309f456fad96592ac72da685 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 02:00:02.766445 17755 ts_tablet_manager.cc:1367] T 71e8030b309f456fad96592ac72da685 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:00:02.767144 17831 tablet_bootstrap.cc:492] T 71e8030b309f456fad96592ac72da685 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 02:00:02.767242 17831 ts_tablet_manager.cc:1367] T 71e8030b309f456fad96592ac72da685 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:00:02.767091 17755 raft_consensus.cc:348] T 71e8030b309f456fad96592ac72da685 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:02.767661 17755 raft_consensus.cc:374] T 71e8030b309f456fad96592ac72da685 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:02.767751 17755 raft_consensus.cc:729] T 71e8030b309f456fad96592ac72da685 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.767724 17831 raft_consensus.cc:348] T 71e8030b309f456fad96592ac72da685 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:02.767836 17831 raft_consensus.cc:374] T 71e8030b309f456fad96592ac72da685 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:02.767884 17831 raft_consensus.cc:729] T 71e8030b309f456fad96592ac72da685 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 02:00:02.768025 17831 consensus_queue.cc:260] T 71e8030b309f456fad96592ac72da685 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:02.768041 17755 consensus_queue.cc:260] T 71e8030b309f456fad96592ac72da685 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:02.770715 17755 ts_tablet_manager.cc:1398] T 71e8030b309f456fad96592ac72da685 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:00:02.772074 17831 ts_tablet_manager.cc:1398] T 71e8030b309f456fad96592ac72da685 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:02.851085 15159 raft_consensus.cc:482] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:02.851282 15159 raft_consensus.cc:504] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:02.851742 15159 leader_election.cc:288] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 02:00:02.852308 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06bffbde7b9f4b31835f61795e6ca32a" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 02:00:02.852339 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06bffbde7b9f4b31835f61795e6ca32a" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 02:00:02.852461 7275 raft_consensus.cc:2454] T 06bffbde7b9f4b31835f61795e6ca32a P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 02:00:02.852459 9493 raft_consensus.cc:2454] T 06bffbde7b9f4b31835f61795e6ca32a P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 02:00:02.852770 7079 leader_election.cc:302] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 02:00:02.853121 15159 raft_consensus.cc:2781] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:02.853189 15159 raft_consensus.cc:482] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:02.853240 15159 raft_consensus.cc:3037] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:02.855250 15159 raft_consensus.cc:504] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:02.855800 15159 leader_election.cc:288] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Requested vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 02:00:02.856328 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06bffbde7b9f4b31835f61795e6ca32a" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 02:00:02.856390 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06bffbde7b9f4b31835f61795e6ca32a" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 02:00:02.856467 9493 raft_consensus.cc:3037] T 06bffbde7b9f4b31835f61795e6ca32a P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:02.856513 7275 raft_consensus.cc:3037] T 06bffbde7b9f4b31835f61795e6ca32a P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:02.857606 9493 raft_consensus.cc:2454] T 06bffbde7b9f4b31835f61795e6ca32a P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 02:00:02.857761 7275 raft_consensus.cc:2454] T 06bffbde7b9f4b31835f61795e6ca32a P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 02:00:02.858060 7078 leader_election.cc:302] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 02:00:02.858250 15159 raft_consensus.cc:2781] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:02.858480 15159 raft_consensus.cc:686] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Becoming Leader. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Running, Role: LEADER
I20240503 02:00:02.858738 15159 consensus_queue.cc:237] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:02.861807 6739 catalog_manager.cc:5484] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:02.874861 17183 raft_consensus.cc:482] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:02.875005 17183 raft_consensus.cc:504] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:02.875509 17183 leader_election.cc:288] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 02:00:02.875962 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e8030b309f456fad96592ac72da685" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 02:00:02.876051 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e8030b309f456fad96592ac72da685" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 02:00:02.876111 9493 raft_consensus.cc:2454] T 71e8030b309f456fad96592ac72da685 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 02:00:02.876191 7145 raft_consensus.cc:2454] T 71e8030b309f456fad96592ac72da685 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 02:00:02.876679 9692 leader_election.cc:302] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 02:00:02.876858 17183 raft_consensus.cc:2781] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:02.876912 17183 raft_consensus.cc:482] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:02.876948 17183 raft_consensus.cc:3037] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:02.878460 17183 raft_consensus.cc:504] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:02.878877 17183 leader_election.cc:288] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Requested vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 02:00:02.879721 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e8030b309f456fad96592ac72da685" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 02:00:02.879866 7145 raft_consensus.cc:3037] T 71e8030b309f456fad96592ac72da685 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:02.881090 7145 raft_consensus.cc:2454] T 71e8030b309f456fad96592ac72da685 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 02:00:02.882227 9692 leader_election.cc:302] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 02:00:02.882411 17183 raft_consensus.cc:2781] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:02.882478 17183 raft_consensus.cc:686] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Becoming Leader. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Running, Role: LEADER
I20240503 02:00:02.882640 17183 consensus_queue.cc:237] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:02.883805 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71e8030b309f456fad96592ac72da685" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 02:00:02.883939 9494 raft_consensus.cc:3037] T 71e8030b309f456fad96592ac72da685 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:02.885080 9494 raft_consensus.cc:2454] T 71e8030b309f456fad96592ac72da685 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 02:00:02.885653 6739 catalog_manager.cc:5484] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: term changed from 0 to 1, leader changed from <none> to db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:02.894913 16988 consensus_queue.cc:1035] T 5097893b51da40059b0ef865d42c4d4a P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:02.905788 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:02.909435 7254 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:00:02.910629 9472 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:02.910831 17147 consensus_queue.cc:1035] T 5097893b51da40059b0ef865d42c4d4a P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:02.915472 6863 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:00:02.919212 9342 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:02.922500 17508 consensus_queue.cc:1035] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:02.931300 7124 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:00:02.943158 6992 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:00:02.973778 9736 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:02.986071 17511 consensus_queue.cc:1035] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:03.020622 6739 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:03.025089 7145 raft_consensus.cc:1261] T 71e8030b309f456fad96592ac72da685 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:03.025288 9493 raft_consensus.cc:1261] T 71e8030b309f456fad96592ac72da685 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:03.025913 17939 consensus_queue.cc:1035] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:03.026126 16785 consensus_queue.cc:1035] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:03.028379 17785 tablet.cc:1722] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:03.028717 17785 tablet.cc:1620] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 02:00:03.031234 17780 mvcc.cc:205] Tried to move back new op lower bound from 7023417765986799616 to 7023417765408747520. Current Snapshot: MvccSnapshot[applied={T|T < 7023417765986799616}]
I20240503 02:00:03.032655 6733 catalog_manager.cc:5576] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471) does not have the latest schema for tablet 71e8030b309f456fad96592ac72da685 (table rebalancer_dynamic_table [id=003cb51a186e417d844491ae5dee69f5]). Expected version 1 got 0
I20240503 02:00:03.034595 6739 catalog_manager.cc:5576] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979) does not have the latest schema for tablet 71e8030b309f456fad96592ac72da685 (table rebalancer_dynamic_table [id=003cb51a186e417d844491ae5dee69f5]). Expected version 1 got 0
I20240503 02:00:03.035404 17962 tablet.cc:1722] T 71e8030b309f456fad96592ac72da685 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:03.035782 17962 tablet.cc:1620] T 71e8030b309f456fad96592ac72da685 P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 02:00:03.039328 6740 catalog_manager.cc:5576] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979) does not have the latest schema for tablet 71e8030b309f456fad96592ac72da685 (table rebalancer_dynamic_table [id=003cb51a186e417d844491ae5dee69f5]). Expected version 1 got 0
I20240503 02:00:03.040736 6733 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=003cb51a186e417d844491ae5dee69f5] alter complete (version 1)
I20240503 02:00:03.042145 17963 tablet.cc:1722] T 71e8030b309f456fad96592ac72da685 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:03.042706 17963 tablet.cc:1620] T 71e8030b309f456fad96592ac72da685 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: MemRowSet was empty: no flush needed.
I20240503 02:00:03.058137 9625 raft_consensus.cc:1261] T fd29f3b059874b019153b64ef8aaaa48 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 2 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:00:03.058580 16988 consensus_queue.cc:1035] T fd29f3b059874b019153b64ef8aaaa48 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:03.148474 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:03.161779 17785 tablet.cc:1722] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:03.162243 17785 tablet.cc:1620] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191: MemRowSet was empty: no flush needed.
I20240503 02:00:03.164798 17962 tablet.cc:1722] T 71e8030b309f456fad96592ac72da685 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:03.165112 17962 tablet.cc:1620] T 71e8030b309f456fad96592ac72da685 P bb5364d730cb4489a0bfc6c270b7450a: MemRowSet was empty: no flush needed.
I20240503 02:00:03.177734 6737 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=003cb51a186e417d844491ae5dee69f5] alter complete (version 2)
I20240503 02:00:03.184005 17963 tablet.cc:1722] T 71e8030b309f456fad96592ac72da685 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:03.184697 17963 tablet.cc:1620] T 71e8030b309f456fad96592ac72da685 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: MemRowSet was empty: no flush needed.
I20240503 02:00:03.196249 7014 raft_consensus.cc:1261] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:00:03.197306 17147 consensus_queue.cc:1035] T fd29f3b059874b019153b64ef8aaaa48 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:03.200829 6882 raft_consensus.cc:1261] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 2 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:00:03.201817 16988 consensus_queue.cc:1035] T fd29f3b059874b019153b64ef8aaaa48 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:03.226213 9342 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:03.227792 6992 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:00:03.230890 7254 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:00:03.238495 9736 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:03.250711 7124 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:00:03.257890 6863 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:00:03.266242 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:03.270749 9472 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:03.287118 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:03.295392 17785 tablet.cc:1722] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:03.296577 17962 tablet.cc:1722] T 71e8030b309f456fad96592ac72da685 P bb5364d730cb4489a0bfc6c270b7450a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:03.300374 17963 tablet.cc:1722] T 71e8030b309f456fad96592ac72da685 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:03.304533 6733 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=003cb51a186e417d844491ae5dee69f5] alter complete (version 3)
I20240503 02:00:03.377187 9756 consensus_queue.cc:237] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:03.378804 7145 raft_consensus.cc:1261] T e8cb37039be049cf9c4e53bbcfbd80d9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:03.378840 9625 raft_consensus.cc:1261] T e8cb37039be049cf9c4e53bbcfbd80d9 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:03.379619 17183 consensus_queue.cc:1035] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:03.379962 17183 consensus_queue.cc:1035] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:03.383126 17939 raft_consensus.cc:2932] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:03.385344 7145 raft_consensus.cc:2932] T e8cb37039be049cf9c4e53bbcfbd80d9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:03.385766 9625 raft_consensus.cc:2932] T e8cb37039be049cf9c4e53bbcfbd80d9 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:03.388715 6733 catalog_manager.cc:5484] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: config changed from index 2 to 3, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:03.396498 6739 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:03.399006 6992 tablet_service.cc:1514] Processing DeleteTablet for tablet e8cb37039be049cf9c4e53bbcfbd80d9 with delete_type TABLET_DATA_TOMBSTONED (TS 63331f455be049c1bdd6c23c59ac1bef not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:35240
I20240503 02:00:03.400372 18037 tablet_replica.cc:330] T e8cb37039be049cf9c4e53bbcfbd80d9 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 02:00:03.400615 18037 raft_consensus.cc:2229] T e8cb37039be049cf9c4e53bbcfbd80d9 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:03.400852 18037 raft_consensus.cc:2258] T e8cb37039be049cf9c4e53bbcfbd80d9 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:03.401413 18037 ts_tablet_manager.cc:1875] T e8cb37039be049cf9c4e53bbcfbd80d9 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:03.405704 18037 ts_tablet_manager.cc:1888] T e8cb37039be049cf9c4e53bbcfbd80d9 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:00:03.405887 9605 tablet_service.cc:1467] Processing CreateTablet for tablet aa7f6879b71a412cba659a1c0cbc08d5 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=b40adf713fb540c2a32701de28974432]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:03.405881 6992 tablet_service.cc:1467] Processing CreateTablet for tablet aa7f6879b71a412cba659a1c0cbc08d5 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=b40adf713fb540c2a32701de28974432]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:03.406039 18037 log.cc:1196] T e8cb37039be049cf9c4e53bbcfbd80d9 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/e8cb37039be049cf9c4e53bbcfbd80d9
I20240503 02:00:03.406113 9341 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3be25c6ba341b4a2c5c0007ba2129b (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=b40adf713fb540c2a32701de28974432]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:03.406208 9605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa7f6879b71a412cba659a1c0cbc08d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.406208 6992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa7f6879b71a412cba659a1c0cbc08d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.406400 9341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3be25c6ba341b4a2c5c0007ba2129b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.408565 18044 tablet_bootstrap.cc:492] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 02:00:03.408816 9342 tablet_service.cc:1467] Processing CreateTablet for tablet aa7f6879b71a412cba659a1c0cbc08d5 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=b40adf713fb540c2a32701de28974432]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:03.409080 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa7f6879b71a412cba659a1c0cbc08d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.409142 18045 tablet_bootstrap.cc:492] T aa7f6879b71a412cba659a1c0cbc08d5 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 02:00:03.409183 6739 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:03.409308 6739 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:03.410473 18046 tablet_bootstrap.cc:492] T 1d3be25c6ba341b4a2c5c0007ba2129b P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 02:00:03.410719 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet e8cb37039be049cf9c4e53bbcfbd80d9 (table rebalancer_extra_table_48 [id=b11be10400574bd0a4c4768ad34c1fc6]) successfully deleted
I20240503 02:00:03.411167 18045 tablet_bootstrap.cc:654] T aa7f6879b71a412cba659a1c0cbc08d5 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.411396 18046 tablet_bootstrap.cc:654] T 1d3be25c6ba341b4a2c5c0007ba2129b P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.411505 18044 tablet_bootstrap.cc:654] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.412024 6739 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 71e8030b309f456fad96592ac72da685
I20240503 02:00:03.413090 7124 tablet_service.cc:1514] Processing DeleteTablet for tablet 71e8030b309f456fad96592ac72da685 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:03 UTC) from {username='slave'} at 127.0.0.1:50370
I20240503 02:00:03.413292 6863 tablet_service.cc:1467] Processing CreateTablet for tablet 0c421d2afb394266abce79be05feb8ad (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=b40adf713fb540c2a32701de28974432]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:03.413347 7125 tablet_service.cc:1467] Processing CreateTablet for tablet 0c421d2afb394266abce79be05feb8ad (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=b40adf713fb540c2a32701de28974432]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:03.413586 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c421d2afb394266abce79be05feb8ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.413977 9604 tablet_service.cc:1467] Processing CreateTablet for tablet 0c421d2afb394266abce79be05feb8ad (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=b40adf713fb540c2a32701de28974432]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:03.414245 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c421d2afb394266abce79be05feb8ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.414376 9604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c421d2afb394266abce79be05feb8ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.414547 9472 tablet_service.cc:1514] Processing DeleteTablet for tablet 71e8030b309f456fad96592ac72da685 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:03 UTC) from {username='slave'} at 127.0.0.1:54678
I20240503 02:00:03.414620 9736 tablet_service.cc:1514] Processing DeleteTablet for tablet 71e8030b309f456fad96592ac72da685 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:03 UTC) from {username='slave'} at 127.0.0.1:48152
I20240503 02:00:03.415028 9605 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3be25c6ba341b4a2c5c0007ba2129b (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=b40adf713fb540c2a32701de28974432]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:03.415246 9605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3be25c6ba341b4a2c5c0007ba2129b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.415751 18044 tablet_bootstrap.cc:492] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 02:00:03.415855 18044 ts_tablet_manager.cc:1367] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:03.416388 18044 raft_consensus.cc:348] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:03.416499 18044 raft_consensus.cc:374] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.416533 18044 raft_consensus.cc:729] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.416750 16780 consensus_queue.cc:1035] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:03.416785 18044 consensus_queue.cc:260] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:03.418125 18044 ts_tablet_manager.cc:1398] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:03.418800 18045 tablet_bootstrap.cc:492] T aa7f6879b71a412cba659a1c0cbc08d5 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 02:00:03.418891 18045 ts_tablet_manager.cc:1367] T aa7f6879b71a412cba659a1c0cbc08d5 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240503 02:00:03.419446 18049 tablet_replica.cc:330] T 71e8030b309f456fad96592ac72da685 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 02:00:03.419602 18049 raft_consensus.cc:2229] T 71e8030b309f456fad96592ac72da685 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:03.419678 18045 raft_consensus.cc:348] T aa7f6879b71a412cba659a1c0cbc08d5 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:03.419780 18045 raft_consensus.cc:374] T aa7f6879b71a412cba659a1c0cbc08d5 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.419814 18045 raft_consensus.cc:729] T aa7f6879b71a412cba659a1c0cbc08d5 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.419813 18049 raft_consensus.cc:2258] T 71e8030b309f456fad96592ac72da685 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:03.419718 6992 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3be25c6ba341b4a2c5c0007ba2129b (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=b40adf713fb540c2a32701de28974432]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:03.419941 18045 consensus_queue.cc:260] T aa7f6879b71a412cba659a1c0cbc08d5 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:03.420027 6992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3be25c6ba341b4a2c5c0007ba2129b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.420351 18045 ts_tablet_manager.cc:1398] T aa7f6879b71a412cba659a1c0cbc08d5 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:03.420550 18049 ts_tablet_manager.cc:1875] T 71e8030b309f456fad96592ac72da685 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:03.421991 18050 tablet_replica.cc:330] T 71e8030b309f456fad96592ac72da685 P bb5364d730cb4489a0bfc6c270b7450a: stopping tablet replica
I20240503 02:00:03.422174 18050 raft_consensus.cc:2229] T 71e8030b309f456fad96592ac72da685 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:03.422284 18045 tablet_bootstrap.cc:492] T 0c421d2afb394266abce79be05feb8ad P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 02:00:03.422305 18050 raft_consensus.cc:2258] T 71e8030b309f456fad96592ac72da685 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:03.423866 18045 tablet_bootstrap.cc:654] T 0c421d2afb394266abce79be05feb8ad P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.424185 18050 ts_tablet_manager.cc:1875] T 71e8030b309f456fad96592ac72da685 P bb5364d730cb4489a0bfc6c270b7450a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:03.420562 18052 tablet_replica.cc:330] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191: stopping tablet replica
I20240503 02:00:03.428220 18052 raft_consensus.cc:2229] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:03.428540 18052 raft_consensus.cc:2258] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:03.430272 18052 ts_tablet_manager.cc:1875] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:03.431831 16780 consensus_queue.cc:1035] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:03.433231 18044 tablet_bootstrap.cc:492] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 02:00:03.436699 18045 tablet_bootstrap.cc:492] T 0c421d2afb394266abce79be05feb8ad P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 02:00:03.436905 18045 ts_tablet_manager.cc:1367] T 0c421d2afb394266abce79be05feb8ad P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20240503 02:00:03.437474 18045 raft_consensus.cc:348] T 0c421d2afb394266abce79be05feb8ad P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:03.437728 18045 raft_consensus.cc:374] T 0c421d2afb394266abce79be05feb8ad P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.437815 18045 raft_consensus.cc:729] T 0c421d2afb394266abce79be05feb8ad P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.437958 18044 tablet_bootstrap.cc:654] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.438001 18045 consensus_queue.cc:260] T 0c421d2afb394266abce79be05feb8ad P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:03.438412 18045 ts_tablet_manager.cc:1398] T 0c421d2afb394266abce79be05feb8ad P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:03.438717 18045 tablet_bootstrap.cc:492] T 1d3be25c6ba341b4a2c5c0007ba2129b P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 02:00:03.439157 18046 tablet_bootstrap.cc:492] T 1d3be25c6ba341b4a2c5c0007ba2129b P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 02:00:03.439345 18046 ts_tablet_manager.cc:1367] T 1d3be25c6ba341b4a2c5c0007ba2129b P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.000s
I20240503 02:00:03.440255 18049 ts_tablet_manager.cc:1888] T 71e8030b309f456fad96592ac72da685 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:03.441360 18052 ts_tablet_manager.cc:1888] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:03.442018 18046 raft_consensus.cc:348] T 1d3be25c6ba341b4a2c5c0007ba2129b P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:03.442232 18046 raft_consensus.cc:374] T 1d3be25c6ba341b4a2c5c0007ba2129b P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.442314 18046 raft_consensus.cc:729] T 1d3be25c6ba341b4a2c5c0007ba2129b P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.441753 18045 tablet_bootstrap.cc:654] T 1d3be25c6ba341b4a2c5c0007ba2129b P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.443630 18046 consensus_queue.cc:260] T 1d3be25c6ba341b4a2c5c0007ba2129b P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:03.444423 18046 ts_tablet_manager.cc:1398] T 1d3be25c6ba341b4a2c5c0007ba2129b P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:03.444768 18052 log.cc:1196] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/71e8030b309f456fad96592ac72da685
I20240503 02:00:03.445399 18057 tablet_bootstrap.cc:492] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 02:00:03.447805 18057 tablet_bootstrap.cc:654] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.448416 18046 tablet_bootstrap.cc:492] T aa7f6879b71a412cba659a1c0cbc08d5 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 02:00:03.448570 18044 tablet_bootstrap.cc:492] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 02:00:03.448674 18044 ts_tablet_manager.cc:1367] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240503 02:00:03.448951 18049 log.cc:1196] T 71e8030b309f456fad96592ac72da685 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/71e8030b309f456fad96592ac72da685
I20240503 02:00:03.449223 18044 raft_consensus.cc:348] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:03.449343 18044 raft_consensus.cc:374] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.449370 18044 raft_consensus.cc:729] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.449388 18049 ts_tablet_manager.cc:1909] T 71e8030b309f456fad96592ac72da685 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting consensus metadata
I20240503 02:00:03.451020 18050 ts_tablet_manager.cc:1888] T 71e8030b309f456fad96592ac72da685 P bb5364d730cb4489a0bfc6c270b7450a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:03.451121 18044 consensus_queue.cc:260] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:03.451644 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet 71e8030b309f456fad96592ac72da685 (table rebalancer_dynamic_table_new_name [id=003cb51a186e417d844491ae5dee69f5]) successfully deleted
I20240503 02:00:03.451704 18050 log.cc:1196] T 71e8030b309f456fad96592ac72da685 P bb5364d730cb4489a0bfc6c270b7450a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/71e8030b309f456fad96592ac72da685
I20240503 02:00:03.452100 18050 ts_tablet_manager.cc:1909] T 71e8030b309f456fad96592ac72da685 P bb5364d730cb4489a0bfc6c270b7450a: Deleting consensus metadata
I20240503 02:00:03.452930 6725 catalog_manager.cc:4830] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): tablet 71e8030b309f456fad96592ac72da685 (table rebalancer_dynamic_table_new_name [id=003cb51a186e417d844491ae5dee69f5]) successfully deleted
I20240503 02:00:03.454782 18045 tablet_bootstrap.cc:492] T 1d3be25c6ba341b4a2c5c0007ba2129b P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 02:00:03.455006 18045 ts_tablet_manager.cc:1367] T 1d3be25c6ba341b4a2c5c0007ba2129b P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240503 02:00:03.455614 18045 raft_consensus.cc:348] T 1d3be25c6ba341b4a2c5c0007ba2129b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:03.455778 18052 ts_tablet_manager.cc:1909] T 71e8030b309f456fad96592ac72da685 P db1a0000aaa346f79c0aa4b6c5105191: Deleting consensus metadata
I20240503 02:00:03.455826 18045 raft_consensus.cc:374] T 1d3be25c6ba341b4a2c5c0007ba2129b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.455884 18045 raft_consensus.cc:729] T 1d3be25c6ba341b4a2c5c0007ba2129b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.456140 18045 consensus_queue.cc:260] T 1d3be25c6ba341b4a2c5c0007ba2129b P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:03.457139 18044 ts_tablet_manager.cc:1398] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240503 02:00:03.457386 6726 catalog_manager.cc:4830] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): tablet 71e8030b309f456fad96592ac72da685 (table rebalancer_dynamic_table_new_name [id=003cb51a186e417d844491ae5dee69f5]) successfully deleted
I20240503 02:00:03.457796 18045 ts_tablet_manager.cc:1398] T 1d3be25c6ba341b4a2c5c0007ba2129b P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:03.458182 18046 tablet_bootstrap.cc:654] T aa7f6879b71a412cba659a1c0cbc08d5 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.462118 18054 tablet_bootstrap.cc:492] T 0c421d2afb394266abce79be05feb8ad P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 02:00:03.463665 18054 tablet_bootstrap.cc:654] T 0c421d2afb394266abce79be05feb8ad P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.466125 18054 tablet_bootstrap.cc:492] T 0c421d2afb394266abce79be05feb8ad P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 02:00:03.466250 18054 ts_tablet_manager.cc:1367] T 0c421d2afb394266abce79be05feb8ad P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 02:00:03.466956 18054 raft_consensus.cc:348] T 0c421d2afb394266abce79be05feb8ad P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:03.467085 18054 raft_consensus.cc:374] T 0c421d2afb394266abce79be05feb8ad P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.467114 18054 raft_consensus.cc:729] T 0c421d2afb394266abce79be05feb8ad P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.467239 18054 consensus_queue.cc:260] T 0c421d2afb394266abce79be05feb8ad P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:03.467617 18054 ts_tablet_manager.cc:1398] T 0c421d2afb394266abce79be05feb8ad P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:03.473438 18057 tablet_bootstrap.cc:492] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 02:00:03.473592 18057 ts_tablet_manager.cc:1367] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.000s
I20240503 02:00:03.474099 18057 raft_consensus.cc:348] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:03.474249 18057 raft_consensus.cc:374] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.474283 18057 raft_consensus.cc:729] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.474418 18057 consensus_queue.cc:260] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:03.474763 18057 ts_tablet_manager.cc:1398] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:03.478179 18046 tablet_bootstrap.cc:492] T aa7f6879b71a412cba659a1c0cbc08d5 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 02:00:03.478307 18046 ts_tablet_manager.cc:1367] T aa7f6879b71a412cba659a1c0cbc08d5 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.000s
I20240503 02:00:03.478981 18046 raft_consensus.cc:348] T aa7f6879b71a412cba659a1c0cbc08d5 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:03.479221 18046 raft_consensus.cc:374] T aa7f6879b71a412cba659a1c0cbc08d5 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.479295 18046 raft_consensus.cc:729] T aa7f6879b71a412cba659a1c0cbc08d5 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.479446 18046 consensus_queue.cc:260] T aa7f6879b71a412cba659a1c0cbc08d5 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:03.481189 18046 ts_tablet_manager.cc:1398] T aa7f6879b71a412cba659a1c0cbc08d5 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:03.520197 6732 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:03.529865 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dad930eecba845c9b272715bd0333ecc]), partition=
I20240503 02:00:03.530206 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ce2a6629ecf4fbbaf7c7d6c2f15ef76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.532753 18046 tablet_bootstrap.cc:492] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 02:00:03.533025 6992 tablet_service.cc:1467] Processing CreateTablet for tablet 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dad930eecba845c9b272715bd0333ecc]), partition=
I20240503 02:00:03.533285 6992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ce2a6629ecf4fbbaf7c7d6c2f15ef76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.533922 18046 tablet_bootstrap.cc:654] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.535581 18046 tablet_bootstrap.cc:492] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 02:00:03.535687 18046 ts_tablet_manager.cc:1367] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:03.536023 7125 tablet_service.cc:1467] Processing CreateTablet for tablet 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dad930eecba845c9b272715bd0333ecc]), partition=
I20240503 02:00:03.536258 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ce2a6629ecf4fbbaf7c7d6c2f15ef76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:03.536240 18046 raft_consensus.cc:348] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:03.536356 18046 raft_consensus.cc:374] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.536397 18046 raft_consensus.cc:729] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.536540 18046 consensus_queue.cc:260] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:03.536844 18046 ts_tablet_manager.cc:1398] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:03.538208 18054 tablet_bootstrap.cc:492] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 02:00:03.538445 18044 tablet_bootstrap.cc:492] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 02:00:03.539212 18054 tablet_bootstrap.cc:654] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.539379 18044 tablet_bootstrap.cc:654] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:03.540594 18054 tablet_bootstrap.cc:492] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 02:00:03.540674 18054 ts_tablet_manager.cc:1367] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:03.540993 18044 tablet_bootstrap.cc:492] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 02:00:03.541075 18044 ts_tablet_manager.cc:1367] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:03.541136 18054 raft_consensus.cc:348] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:03.541257 18054 raft_consensus.cc:374] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.541292 18054 raft_consensus.cc:729] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.541388 18054 consensus_queue.cc:260] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:03.541512 18044 raft_consensus.cc:348] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:03.541623 18044 raft_consensus.cc:374] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:03.541669 18044 raft_consensus.cc:729] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 02:00:03.541689 18054 ts_tablet_manager.cc:1398] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:03.541770 18044 consensus_queue.cc:260] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:03.542024 18044 ts_tablet_manager.cc:1398] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:03.542120 17511 raft_consensus.cc:482] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:03.542183 17511 raft_consensus.cc:504] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:03.542482 17511 leader_election.cc:288] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:03.544313 9361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3be25c6ba341b4a2c5c0007ba2129b" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 02:00:03.544461 9361 raft_consensus.cc:2454] T 1d3be25c6ba341b4a2c5c0007ba2129b P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 02:00:03.544843 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3be25c6ba341b4a2c5c0007ba2129b" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 02:00:03.544978 9625 raft_consensus.cc:2454] T 1d3be25c6ba341b4a2c5c0007ba2129b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 02:00:03.545301 6949 leader_election.cc:302] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 02:00:03.545749 16636 raft_consensus.cc:2781] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:03.545809 16636 raft_consensus.cc:482] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:03.545848 16636 raft_consensus.cc:3037] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:03.547070 16636 raft_consensus.cc:504] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:03.547439 16636 leader_election.cc:288] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:03.549782 9361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3be25c6ba341b4a2c5c0007ba2129b" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 02:00:03.549924 9361 raft_consensus.cc:3037] T 1d3be25c6ba341b4a2c5c0007ba2129b P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:03.550027 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3be25c6ba341b4a2c5c0007ba2129b" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 02:00:03.550161 9625 raft_consensus.cc:3037] T 1d3be25c6ba341b4a2c5c0007ba2129b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:03.551057 9361 raft_consensus.cc:2454] T 1d3be25c6ba341b4a2c5c0007ba2129b P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 02:00:03.551239 9625 raft_consensus.cc:2454] T 1d3be25c6ba341b4a2c5c0007ba2129b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 02:00:03.551364 6949 leader_election.cc:302] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 02:00:03.551719 16636 raft_consensus.cc:2781] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:03.551828 16636 raft_consensus.cc:686] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 02:00:03.551965 16636 consensus_queue.cc:237] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:03.556043 6740 catalog_manager.cc:5484] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:03.645764 18000 raft_consensus.cc:482] T aa7f6879b71a412cba659a1c0cbc08d5 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:03.645884 18000 raft_consensus.cc:504] T aa7f6879b71a412cba659a1c0cbc08d5 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:03.646239 18000 leader_election.cc:288] T aa7f6879b71a412cba659a1c0cbc08d5 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 02:00:03.646638 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7f6879b71a412cba659a1c0cbc08d5" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 02:00:03.646847 9625 raft_consensus.cc:2454] T aa7f6879b71a412cba659a1c0cbc08d5 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 02:00:03.646842 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7f6879b71a412cba659a1c0cbc08d5" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 02:00:03.646973 7014 raft_consensus.cc:2454] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 0.
I20240503 02:00:03.647141 9298 leader_election.cc:302] T aa7f6879b71a412cba659a1c0cbc08d5 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 02:00:03.647348 18000 raft_consensus.cc:2781] T aa7f6879b71a412cba659a1c0cbc08d5 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:03.647397 18000 raft_consensus.cc:482] T aa7f6879b71a412cba659a1c0cbc08d5 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:03.647434 18000 raft_consensus.cc:3037] T aa7f6879b71a412cba659a1c0cbc08d5 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:03.648389 18000 raft_consensus.cc:504] T aa7f6879b71a412cba659a1c0cbc08d5 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:03.648965 18000 leader_election.cc:288] T aa7f6879b71a412cba659a1c0cbc08d5 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 02:00:03.649075 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7f6879b71a412cba659a1c0cbc08d5" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 02:00:03.649163 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa7f6879b71a412cba659a1c0cbc08d5" candidate_uuid: "12e8c683db054df3b75c6285b98a5ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 02:00:03.649194 7014 raft_consensus.cc:3037] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:03.649405 9625 raft_consensus.cc:3037] T aa7f6879b71a412cba659a1c0cbc08d5 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:03.650924 7014 raft_consensus.cc:2454] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 02:00:03.650979 9625 raft_consensus.cc:2454] T aa7f6879b71a412cba659a1c0cbc08d5 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e8c683db054df3b75c6285b98a5ad8 in term 1.
I20240503 02:00:03.651301 9298 leader_election.cc:302] T aa7f6879b71a412cba659a1c0cbc08d5 P 12e8c683db054df3b75c6285b98a5ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 02:00:03.651491 18000 raft_consensus.cc:2781] T aa7f6879b71a412cba659a1c0cbc08d5 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:03.651710 18000 raft_consensus.cc:686] T aa7f6879b71a412cba659a1c0cbc08d5 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Running, Role: LEADER
I20240503 02:00:03.651873 18000 consensus_queue.cc:237] T aa7f6879b71a412cba659a1c0cbc08d5 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:03.654979 6733 catalog_manager.cc:5484] T aa7f6879b71a412cba659a1c0cbc08d5 P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133). New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:03.727970 18060 raft_consensus.cc:482] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:03.728127 18060 raft_consensus.cc:504] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:03.729056 18060 leader_election.cc:288] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:03.729519 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c421d2afb394266abce79be05feb8ad" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 02:00:03.729684 7143 raft_consensus.cc:2454] T 0c421d2afb394266abce79be05feb8ad P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 02:00:03.730410 6818 leader_election.cc:302] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 02:00:03.730620 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c421d2afb394266abce79be05feb8ad" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 02:00:03.730902 9625 raft_consensus.cc:2454] T 0c421d2afb394266abce79be05feb8ad P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 02:00:03.730911 18060 raft_consensus.cc:2781] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:03.731000 18060 raft_consensus.cc:482] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:03.731032 18060 raft_consensus.cc:3037] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:03.732213 18060 raft_consensus.cc:504] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:03.732620 18060 leader_election.cc:288] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Requested vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:03.733517 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c421d2afb394266abce79be05feb8ad" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 02:00:03.733683 7145 raft_consensus.cc:3037] T 0c421d2afb394266abce79be05feb8ad P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:03.733649 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c421d2afb394266abce79be05feb8ad" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 02:00:03.733765 9625 raft_consensus.cc:3037] T 0c421d2afb394266abce79be05feb8ad P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:03.734859 7145 raft_consensus.cc:2454] T 0c421d2afb394266abce79be05feb8ad P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 02:00:03.734859 9625 raft_consensus.cc:2454] T 0c421d2afb394266abce79be05feb8ad P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 02:00:03.735222 6818 leader_election.cc:302] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 02:00:03.735458 18060 raft_consensus.cc:2781] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:03.735847 18060 raft_consensus.cc:686] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 02:00:03.736369 18060 consensus_queue.cc:237] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:03.739842 6733 catalog_manager.cc:5484] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:03.748147 15159 raft_consensus.cc:482] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:03.748406 15159 raft_consensus.cc:504] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:03.750504 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ce2a6629ecf4fbbaf7c7d6c2f15ef76" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 02:00:03.750648 9361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ce2a6629ecf4fbbaf7c7d6c2f15ef76" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 02:00:03.750734 7014 raft_consensus.cc:2454] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 02:00:03.750824 9361 raft_consensus.cc:2454] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 02:00:03.751261 7078 leader_election.cc:302] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 02:00:03.751528 16780 raft_consensus.cc:2781] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:03.751600 16780 raft_consensus.cc:482] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:03.751675 16780 raft_consensus.cc:3037] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:03.753355 16780 raft_consensus.cc:504] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:03.753813 16780 leader_election.cc:288] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Requested vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 02:00:03.754405 9361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ce2a6629ecf4fbbaf7c7d6c2f15ef76" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 02:00:03.754549 9361 raft_consensus.cc:3037] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:03.755587 9361 raft_consensus.cc:2454] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 02:00:03.755970 7080 leader_election.cc:302] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 02:00:03.757303 16780 raft_consensus.cc:2781] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:03.757450 16780 raft_consensus.cc:686] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Becoming Leader. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Running, Role: LEADER
I20240503 02:00:03.757439 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ce2a6629ecf4fbbaf7c7d6c2f15ef76" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 02:00:03.757608 7014 raft_consensus.cc:3037] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:03.757622 16780 consensus_queue.cc:237] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:03.758845 7014 raft_consensus.cc:2454] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 02:00:03.751591 15159 leader_election.cc:288] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 02:00:03.762159 6733 catalog_manager.cc:5484] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:03.768796 9342 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:03.774428 6863 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:03.794090 7254 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:00:03.795097 7125 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:03.813984 6992 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:03.827497 9736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:00:03.842346 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:03.863070 9472 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:03.911798 6733 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:03.916226 7014 raft_consensus.cc:1261] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:03.916707 9361 raft_consensus.cc:1261] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:03.918859 18115 consensus_queue.cc:1035] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:03.919318 18115 consensus_queue.cc:1035] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:03.923633 6735 catalog_manager.cc:5576] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231) does not have the latest schema for tablet 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 (table rebalancer_dynamic_table [id=dad930eecba845c9b272715bd0333ecc]). Expected version 1 got 0
I20240503 02:00:03.927223 6735 catalog_manager.cc:5576] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387) does not have the latest schema for tablet 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 (table rebalancer_dynamic_table [id=dad930eecba845c9b272715bd0333ecc]). Expected version 1 got 0
I20240503 02:00:03.927629 18134 tablet.cc:1722] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:03.928010 18134 tablet.cc:1620] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: MemRowSet was empty: no flush needed.
I20240503 02:00:03.928617 18135 tablet.cc:1722] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 63331f455be049c1bdd6c23c59ac1bef: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:03.928936 18135 tablet.cc:1620] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 63331f455be049c1bdd6c23c59ac1bef: MemRowSet was empty: no flush needed.
I20240503 02:00:03.933319 18132 mvcc.cc:205] Tried to move back new op lower bound from 7023417769636749312 to 7023417768990830592. Current Snapshot: MvccSnapshot[applied={T|T < 7023417769636749312}]
I20240503 02:00:03.933194 18136 tablet.cc:1722] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:03.935566 18136 tablet.cc:1620] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 02:00:03.937469 6732 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=dad930eecba845c9b272715bd0333ecc] alter complete (version 1)
I20240503 02:00:03.937659 6733 catalog_manager.cc:5576] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387) does not have the latest schema for tablet 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 (table rebalancer_dynamic_table [id=dad930eecba845c9b272715bd0333ecc]). Expected version 1 got 0
I20240503 02:00:03.955632 9361 consensus_queue.cc:237] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:03.957077 9625 raft_consensus.cc:1261] T 22b6ec13c4e04d6cbade150f0c644e52 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:03.957216 7143 raft_consensus.cc:1261] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:03.957489 16988 consensus_queue.cc:1035] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:03.958010 16988 consensus_queue.cc:1035] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:03.960177 16988 raft_consensus.cc:2932] T 22b6ec13c4e04d6cbade150f0c644e52 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:03.960680 9625 raft_consensus.cc:2932] T 22b6ec13c4e04d6cbade150f0c644e52 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:03.965713 6733 catalog_manager.cc:5484] T 22b6ec13c4e04d6cbade150f0c644e52 P b7062fbbc2b54fff9e99ec4ea5562c76 reported cstate change: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:03.971869 9472 tablet_service.cc:1514] Processing DeleteTablet for tablet 22b6ec13c4e04d6cbade150f0c644e52 with delete_type TABLET_DATA_TOMBSTONED (TS bb5364d730cb4489a0bfc6c270b7450a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54678
I20240503 02:00:03.971791 7143 raft_consensus.cc:2932] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:03.972378 18151 tablet_replica.cc:330] T 22b6ec13c4e04d6cbade150f0c644e52 P bb5364d730cb4489a0bfc6c270b7450a: stopping tablet replica
I20240503 02:00:03.972815 18151 raft_consensus.cc:2229] T 22b6ec13c4e04d6cbade150f0c644e52 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:03.973204 18151 raft_consensus.cc:2258] T 22b6ec13c4e04d6cbade150f0c644e52 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:03.974635 18151 ts_tablet_manager.cc:1875] T 22b6ec13c4e04d6cbade150f0c644e52 P bb5364d730cb4489a0bfc6c270b7450a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:03.978986 18151 ts_tablet_manager.cc:1888] T 22b6ec13c4e04d6cbade150f0c644e52 P bb5364d730cb4489a0bfc6c270b7450a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:00:03.979260 18151 log.cc:1196] T 22b6ec13c4e04d6cbade150f0c644e52 P bb5364d730cb4489a0bfc6c270b7450a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/22b6ec13c4e04d6cbade150f0c644e52
I20240503 02:00:03.980221 6725 catalog_manager.cc:4830] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): tablet 22b6ec13c4e04d6cbade150f0c644e52 (table rebalancer_extra_table_28 [id=9a04da30c0934e269de5ed55fe118bee]) successfully deleted
I20240503 02:00:04.009321 17508 consensus_queue.cc:1035] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:04.014871 17508 consensus_queue.cc:1035] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:04.042270 16988 consensus_queue.cc:1035] T aa7f6879b71a412cba659a1c0cbc08d5 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:04.048173 6734 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:04.059557 18000 consensus_queue.cc:1035] T aa7f6879b71a412cba659a1c0cbc08d5 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:04.060525 18134 tablet.cc:1722] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:04.060889 18134 tablet.cc:1620] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: MemRowSet was empty: no flush needed.
I20240503 02:00:04.065536 18136 tablet.cc:1722] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:04.066025 18136 tablet.cc:1620] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 02:00:04.067310 18135 tablet.cc:1722] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 63331f455be049c1bdd6c23c59ac1bef: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:04.067621 18135 tablet.cc:1620] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 63331f455be049c1bdd6c23c59ac1bef: MemRowSet was empty: no flush needed.
I20240503 02:00:04.073027 6735 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=dad930eecba845c9b272715bd0333ecc] alter complete (version 2)
I20240503 02:00:04.127990 18060 consensus_queue.cc:1035] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:04.146243 18060 consensus_queue.cc:1035] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:04.185950 6735 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:04.192620 18134 tablet.cc:1722] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:04.193725 18136 tablet.cc:1722] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:04.199008 18135 tablet.cc:1722] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 63331f455be049c1bdd6c23c59ac1bef: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:04.205511 6735 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=dad930eecba845c9b272715bd0333ecc] alter complete (version 3)
I20240503 02:00:04.231612 9342 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:04.233105 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:04.236414 7125 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:04.237063 7254 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 02:00:04.237429 9472 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:04.247143 6735 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:04.257018 6992 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:04.258433 7254 tablet_service.cc:1467] Processing CreateTablet for tablet 743d87c1c6ad4f7585514f1adb350ae1 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=4b423fe4697f4f2b99ff134ac4c2c23d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:04.258837 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 743d87c1c6ad4f7585514f1adb350ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.259231 7125 tablet_service.cc:1467] Processing CreateTablet for tablet 743d87c1c6ad4f7585514f1adb350ae1 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=4b423fe4697f4f2b99ff134ac4c2c23d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:04.259488 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 743d87c1c6ad4f7585514f1adb350ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.260113 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 743d87c1c6ad4f7585514f1adb350ae1 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=4b423fe4697f4f2b99ff134ac4c2c23d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:04.260521 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 743d87c1c6ad4f7585514f1adb350ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.262400 6863 tablet_service.cc:1467] Processing CreateTablet for tablet f18922f489de49d5b4767afc9ca1b2ed (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=4b423fe4697f4f2b99ff134ac4c2c23d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:04.262754 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f18922f489de49d5b4767afc9ca1b2ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.262861 9472 tablet_service.cc:1467] Processing CreateTablet for tablet f18922f489de49d5b4767afc9ca1b2ed (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=4b423fe4697f4f2b99ff134ac4c2c23d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:04.263132 9472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f18922f489de49d5b4767afc9ca1b2ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.263556 7254 tablet_service.cc:1467] Processing CreateTablet for tablet 389caa08cfcf48a0b91aa5d083cc7d18 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=4b423fe4697f4f2b99ff134ac4c2c23d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:04.263823 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 389caa08cfcf48a0b91aa5d083cc7d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.266960 9605 tablet_service.cc:1467] Processing CreateTablet for tablet 389caa08cfcf48a0b91aa5d083cc7d18 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=4b423fe4697f4f2b99ff134ac4c2c23d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:04.267189 9735 tablet_service.cc:1467] Processing CreateTablet for tablet 389caa08cfcf48a0b91aa5d083cc7d18 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=4b423fe4697f4f2b99ff134ac4c2c23d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:04.267189 9736 tablet_service.cc:1467] Processing CreateTablet for tablet f18922f489de49d5b4767afc9ca1b2ed (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=4b423fe4697f4f2b99ff134ac4c2c23d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:04.267374 9605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 389caa08cfcf48a0b91aa5d083cc7d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.267537 9735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 389caa08cfcf48a0b91aa5d083cc7d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.267776 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f18922f489de49d5b4767afc9ca1b2ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.270807 18230 tablet_bootstrap.cc:492] T 389caa08cfcf48a0b91aa5d083cc7d18 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 02:00:04.272074 18230 tablet_bootstrap.cc:654] T 389caa08cfcf48a0b91aa5d083cc7d18 P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.273911 18230 tablet_bootstrap.cc:492] T 389caa08cfcf48a0b91aa5d083cc7d18 P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 02:00:04.274005 18230 ts_tablet_manager.cc:1367] T 389caa08cfcf48a0b91aa5d083cc7d18 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:04.274446 18230 raft_consensus.cc:348] T 389caa08cfcf48a0b91aa5d083cc7d18 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:04.274580 18230 raft_consensus.cc:374] T 389caa08cfcf48a0b91aa5d083cc7d18 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.274609 18230 raft_consensus.cc:729] T 389caa08cfcf48a0b91aa5d083cc7d18 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.274741 18230 consensus_queue.cc:260] T 389caa08cfcf48a0b91aa5d083cc7d18 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:04.274977 18230 ts_tablet_manager.cc:1398] T 389caa08cfcf48a0b91aa5d083cc7d18 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:04.275311 18226 tablet_bootstrap.cc:492] T 743d87c1c6ad4f7585514f1adb350ae1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 02:00:04.275537 18225 tablet_bootstrap.cc:492] T 743d87c1c6ad4f7585514f1adb350ae1 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 02:00:04.277806 18225 tablet_bootstrap.cc:654] T 743d87c1c6ad4f7585514f1adb350ae1 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.278107 18229 tablet_bootstrap.cc:492] T f18922f489de49d5b4767afc9ca1b2ed P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 02:00:04.279356 18229 tablet_bootstrap.cc:654] T f18922f489de49d5b4767afc9ca1b2ed P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.279775 18226 tablet_bootstrap.cc:654] T 743d87c1c6ad4f7585514f1adb350ae1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.283154 9736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:00:04.283563 18226 tablet_bootstrap.cc:492] T 743d87c1c6ad4f7585514f1adb350ae1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 02:00:04.283744 18226 ts_tablet_manager.cc:1367] T 743d87c1c6ad4f7585514f1adb350ae1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:00:04.284343 18226 raft_consensus.cc:348] T 743d87c1c6ad4f7585514f1adb350ae1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:04.284937 18225 tablet_bootstrap.cc:492] T 743d87c1c6ad4f7585514f1adb350ae1 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 02:00:04.285084 18225 ts_tablet_manager.cc:1367] T 743d87c1c6ad4f7585514f1adb350ae1 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240503 02:00:04.285622 18225 raft_consensus.cc:348] T 743d87c1c6ad4f7585514f1adb350ae1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:04.285830 18226 raft_consensus.cc:374] T 743d87c1c6ad4f7585514f1adb350ae1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.285915 18226 raft_consensus.cc:729] T 743d87c1c6ad4f7585514f1adb350ae1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.286105 18226 consensus_queue.cc:260] T 743d87c1c6ad4f7585514f1adb350ae1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:04.287598 18226 ts_tablet_manager.cc:1398] T 743d87c1c6ad4f7585514f1adb350ae1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:00:04.287784 18225 raft_consensus.cc:374] T 743d87c1c6ad4f7585514f1adb350ae1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.287843 18225 raft_consensus.cc:729] T 743d87c1c6ad4f7585514f1adb350ae1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.288343 18225 consensus_queue.cc:260] T 743d87c1c6ad4f7585514f1adb350ae1 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:04.288980 18225 ts_tablet_manager.cc:1398] T 743d87c1c6ad4f7585514f1adb350ae1 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:00:04.289261 18225 tablet_bootstrap.cc:492] T 389caa08cfcf48a0b91aa5d083cc7d18 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 02:00:04.302037 6861 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:04.308843 18225 tablet_bootstrap.cc:654] T 389caa08cfcf48a0b91aa5d083cc7d18 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.310117 6735 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:04.310348 6735 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:04.311048 18229 tablet_bootstrap.cc:492] T f18922f489de49d5b4767afc9ca1b2ed P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 02:00:04.314052 18229 ts_tablet_manager.cc:1367] T f18922f489de49d5b4767afc9ca1b2ed P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.000s
I20240503 02:00:04.315973 6735 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 4ce2a6629ecf4fbbaf7c7d6c2f15ef76
I20240503 02:00:04.317514 9342 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:04 UTC) from {username='slave'} at 127.0.0.1:36040
I20240503 02:00:04.317703 7125 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:04 UTC) from {username='slave'} at 127.0.0.1:50370
I20240503 02:00:04.319726 18229 raft_consensus.cc:348] T f18922f489de49d5b4767afc9ca1b2ed P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:04.319948 18229 raft_consensus.cc:374] T f18922f489de49d5b4767afc9ca1b2ed P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.320030 18229 raft_consensus.cc:729] T f18922f489de49d5b4767afc9ca1b2ed P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.320231 18229 consensus_queue.cc:260] T f18922f489de49d5b4767afc9ca1b2ed P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:04.321554 6992 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:04 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 02:00:04.324800 18229 ts_tablet_manager.cc:1398] T f18922f489de49d5b4767afc9ca1b2ed P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240503 02:00:04.327200 18229 tablet_bootstrap.cc:492] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 02:00:04.329159 18229 tablet_bootstrap.cc:654] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.339449 18227 tablet_bootstrap.cc:492] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 02:00:04.342742 18227 tablet_bootstrap.cc:654] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.347108 18228 tablet_bootstrap.cc:492] T f18922f489de49d5b4767afc9ca1b2ed P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 02:00:04.358232 18237 tablet_replica.cc:330] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 02:00:04.361047 18237 raft_consensus.cc:2229] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:04.361486 18237 raft_consensus.cc:2258] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:04.362231 18236 tablet_replica.cc:330] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 12e8c683db054df3b75c6285b98a5ad8: stopping tablet replica
I20240503 02:00:04.362452 18236 raft_consensus.cc:2229] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:04.362685 18236 raft_consensus.cc:2258] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:04.363360 18236 ts_tablet_manager.cc:1875] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 12e8c683db054df3b75c6285b98a5ad8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:04.363955 18237 ts_tablet_manager.cc:1875] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:04.367175 18236 ts_tablet_manager.cc:1888] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 12e8c683db054df3b75c6285b98a5ad8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:04.367358 18236 log.cc:1196] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 12e8c683db054df3b75c6285b98a5ad8: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/4ce2a6629ecf4fbbaf7c7d6c2f15ef76
I20240503 02:00:04.367700 18236 ts_tablet_manager.cc:1909] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 12e8c683db054df3b75c6285b98a5ad8: Deleting consensus metadata
I20240503 02:00:04.368252 18229 tablet_bootstrap.cc:492] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 02:00:04.368423 18229 ts_tablet_manager.cc:1367] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.000s
I20240503 02:00:04.368716 6727 catalog_manager.cc:4830] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): tablet 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 (table rebalancer_dynamic_table_new_name [id=dad930eecba845c9b272715bd0333ecc]) successfully deleted
I20240503 02:00:04.372316 18229 raft_consensus.cc:348] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:04.373237 18229 raft_consensus.cc:374] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.374429 18229 raft_consensus.cc:729] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.374737 18228 tablet_bootstrap.cc:654] T f18922f489de49d5b4767afc9ca1b2ed P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.374843 18229 consensus_queue.cc:260] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:04.375377 18229 ts_tablet_manager.cc:1398] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:00:04.378070 18238 tablet_replica.cc:330] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 02:00:04.378278 18225 tablet_bootstrap.cc:492] T 389caa08cfcf48a0b91aa5d083cc7d18 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 02:00:04.378296 18238 raft_consensus.cc:2229] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:04.378398 18225 ts_tablet_manager.cc:1367] T 389caa08cfcf48a0b91aa5d083cc7d18 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.089s user 0.000s sys 0.004s
I20240503 02:00:04.378392 18237 ts_tablet_manager.cc:1888] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:04.378471 18237 log.cc:1196] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/4ce2a6629ecf4fbbaf7c7d6c2f15ef76
I20240503 02:00:04.378589 18238 raft_consensus.cc:2258] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:04.379305 18238 ts_tablet_manager.cc:1875] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:04.380929 18237 ts_tablet_manager.cc:1909] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting consensus metadata
I20240503 02:00:04.381289 18225 raft_consensus.cc:348] T 389caa08cfcf48a0b91aa5d083cc7d18 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:04.381441 18225 raft_consensus.cc:374] T 389caa08cfcf48a0b91aa5d083cc7d18 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.381479 18225 raft_consensus.cc:729] T 389caa08cfcf48a0b91aa5d083cc7d18 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.381532 18227 tablet_bootstrap.cc:492] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 02:00:04.381625 18227 ts_tablet_manager.cc:1367] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.042s user 0.000s sys 0.002s
I20240503 02:00:04.381633 18225 consensus_queue.cc:260] T 389caa08cfcf48a0b91aa5d083cc7d18 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:04.381958 18225 ts_tablet_manager.cc:1398] T 389caa08cfcf48a0b91aa5d083cc7d18 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 02:00:04.382153 18227 raft_consensus.cc:348] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:04.382257 18227 raft_consensus.cc:374] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.382288 18227 raft_consensus.cc:729] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.382411 18227 consensus_queue.cc:260] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:04.382434 18228 tablet_bootstrap.cc:492] T f18922f489de49d5b4767afc9ca1b2ed P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 02:00:04.382581 18228 ts_tablet_manager.cc:1367] T f18922f489de49d5b4767afc9ca1b2ed P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.000s
I20240503 02:00:04.382756 18227 ts_tablet_manager.cc:1398] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:04.382936 18227 tablet_bootstrap.cc:492] T 743d87c1c6ad4f7585514f1adb350ae1 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 02:00:04.386541 18228 raft_consensus.cc:348] T f18922f489de49d5b4767afc9ca1b2ed P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:04.386713 18228 raft_consensus.cc:374] T f18922f489de49d5b4767afc9ca1b2ed P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.386741 18228 raft_consensus.cc:729] T f18922f489de49d5b4767afc9ca1b2ed P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.386868 18228 consensus_queue.cc:260] T f18922f489de49d5b4767afc9ca1b2ed P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:04.387145 18228 ts_tablet_manager.cc:1398] T f18922f489de49d5b4767afc9ca1b2ed P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:00:04.387621 18227 tablet_bootstrap.cc:654] T 743d87c1c6ad4f7585514f1adb350ae1 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.389222 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 (table rebalancer_dynamic_table_new_name [id=dad930eecba845c9b272715bd0333ecc]) successfully deleted
I20240503 02:00:04.391098 18238 ts_tablet_manager.cc:1888] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:04.391269 18238 log.cc:1196] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/4ce2a6629ecf4fbbaf7c7d6c2f15ef76
I20240503 02:00:04.392096 18060 raft_consensus.cc:482] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:04.392239 18060 raft_consensus.cc:504] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:04.392678 18060 leader_election.cc:288] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 02:00:04.393078 18238 ts_tablet_manager.cc:1909] T 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 02:00:04.393139 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f18922f489de49d5b4767afc9ca1b2ed" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 02:00:04.393285 9756 raft_consensus.cc:2454] T f18922f489de49d5b4767afc9ca1b2ed P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 02:00:04.393783 6817 leader_election.cc:302] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 02:00:04.394654 18060 raft_consensus.cc:2781] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:04.394737 18060 raft_consensus.cc:482] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:04.394776 18060 raft_consensus.cc:3037] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.395009 18227 tablet_bootstrap.cc:492] T 743d87c1c6ad4f7585514f1adb350ae1 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 02:00:04.395107 18227 ts_tablet_manager.cc:1367] T 743d87c1c6ad4f7585514f1adb350ae1 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20240503 02:00:04.395705 18227 raft_consensus.cc:348] T 743d87c1c6ad4f7585514f1adb350ae1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:04.395823 18227 raft_consensus.cc:374] T 743d87c1c6ad4f7585514f1adb350ae1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.395853 18227 raft_consensus.cc:729] T 743d87c1c6ad4f7585514f1adb350ae1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.395977 18227 consensus_queue.cc:260] T 743d87c1c6ad4f7585514f1adb350ae1 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:04.396315 18227 ts_tablet_manager.cc:1398] T 743d87c1c6ad4f7585514f1adb350ae1 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:04.396824 18060 raft_consensus.cc:504] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:04.397347 18060 leader_election.cc:288] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Requested vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 02:00:04.397617 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f18922f489de49d5b4767afc9ca1b2ed" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 02:00:04.397825 9493 raft_consensus.cc:2454] T f18922f489de49d5b4767afc9ca1b2ed P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 02:00:04.398852 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f18922f489de49d5b4767afc9ca1b2ed" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 02:00:04.399039 9756 raft_consensus.cc:3037] T f18922f489de49d5b4767afc9ca1b2ed P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.400255 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f18922f489de49d5b4767afc9ca1b2ed" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 02:00:04.400415 9493 raft_consensus.cc:3037] T f18922f489de49d5b4767afc9ca1b2ed P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.400790 9756 raft_consensus.cc:2454] T f18922f489de49d5b4767afc9ca1b2ed P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 02:00:04.401271 6817 leader_election.cc:302] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 02:00:04.401770 18213 raft_consensus.cc:2781] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:04.401856 18213 raft_consensus.cc:686] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 02:00:04.402016 9493 raft_consensus.cc:2454] T f18922f489de49d5b4767afc9ca1b2ed P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 02:00:04.402009 18213 consensus_queue.cc:237] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:04.406915 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet 4ce2a6629ecf4fbbaf7c7d6c2f15ef76 (table rebalancer_dynamic_table_new_name [id=dad930eecba845c9b272715bd0333ecc]) successfully deleted
I20240503 02:00:04.409507 6735 catalog_manager.cc:5484] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:04.414098 16785 raft_consensus.cc:482] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:04.414252 16785 raft_consensus.cc:504] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:04.414759 16785 leader_election.cc:288] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:04.415565 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389caa08cfcf48a0b91aa5d083cc7d18" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 02:00:04.415738 7275 raft_consensus.cc:2454] T 389caa08cfcf48a0b91aa5d083cc7d18 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 02:00:04.415894 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389caa08cfcf48a0b91aa5d083cc7d18" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 02:00:04.416044 9625 raft_consensus.cc:2454] T 389caa08cfcf48a0b91aa5d083cc7d18 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 0.
I20240503 02:00:04.416429 9692 leader_election.cc:302] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 02:00:04.416626 16785 raft_consensus.cc:2781] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:04.416716 16785 raft_consensus.cc:482] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:04.416764 16785 raft_consensus.cc:3037] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.418354 16785 raft_consensus.cc:504] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:04.419133 16785 leader_election.cc:288] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:04.419876 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389caa08cfcf48a0b91aa5d083cc7d18" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 02:00:04.420051 7275 raft_consensus.cc:3037] T 389caa08cfcf48a0b91aa5d083cc7d18 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.420404 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "389caa08cfcf48a0b91aa5d083cc7d18" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 02:00:04.420521 9625 raft_consensus.cc:3037] T 389caa08cfcf48a0b91aa5d083cc7d18 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.421303 7275 raft_consensus.cc:2454] T 389caa08cfcf48a0b91aa5d083cc7d18 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 02:00:04.421593 9625 raft_consensus.cc:2454] T 389caa08cfcf48a0b91aa5d083cc7d18 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 02:00:04.423231 9692 leader_election.cc:302] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7062fbbc2b54fff9e99ec4ea5562c76, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 02:00:04.423446 16785 raft_consensus.cc:2781] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:04.423552 16785 raft_consensus.cc:686] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Becoming Leader. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Running, Role: LEADER
I20240503 02:00:04.423696 16785 consensus_queue.cc:237] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:04.427736 6733 catalog_manager.cc:5484] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: term changed from 0 to 1, leader changed from <none> to db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136). New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:04.427891 6735 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:04.437160 6992 tablet_service.cc:1467] Processing CreateTablet for tablet 61b81445adb34e1e934607d50aa2fcfb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=54e17be95a014be7bd8fd4de99c8a292]), partition=
I20240503 02:00:04.437156 9605 tablet_service.cc:1467] Processing CreateTablet for tablet 61b81445adb34e1e934607d50aa2fcfb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=54e17be95a014be7bd8fd4de99c8a292]), partition=
I20240503 02:00:04.437548 6992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61b81445adb34e1e934607d50aa2fcfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.437662 9605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61b81445adb34e1e934607d50aa2fcfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.440521 18230 tablet_bootstrap.cc:492] T 61b81445adb34e1e934607d50aa2fcfb P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 02:00:04.441869 18230 tablet_bootstrap.cc:654] T 61b81445adb34e1e934607d50aa2fcfb P b7062fbbc2b54fff9e99ec4ea5562c76: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.442878 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 61b81445adb34e1e934607d50aa2fcfb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=54e17be95a014be7bd8fd4de99c8a292]), partition=
I20240503 02:00:04.443185 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61b81445adb34e1e934607d50aa2fcfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:04.443542 18230 tablet_bootstrap.cc:492] T 61b81445adb34e1e934607d50aa2fcfb P b7062fbbc2b54fff9e99ec4ea5562c76: No bootstrap required, opened a new log
I20240503 02:00:04.443655 18230 ts_tablet_manager.cc:1367] T 61b81445adb34e1e934607d50aa2fcfb P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:04.444509 18230 raft_consensus.cc:348] T 61b81445adb34e1e934607d50aa2fcfb P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:04.444633 18230 raft_consensus.cc:374] T 61b81445adb34e1e934607d50aa2fcfb P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.444658 18230 raft_consensus.cc:729] T 61b81445adb34e1e934607d50aa2fcfb P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.444795 18230 consensus_queue.cc:260] T 61b81445adb34e1e934607d50aa2fcfb P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:04.445163 18230 ts_tablet_manager.cc:1398] T 61b81445adb34e1e934607d50aa2fcfb P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:04.447921 18256 tablet_bootstrap.cc:492] T 61b81445adb34e1e934607d50aa2fcfb P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 02:00:04.449558 18256 tablet_bootstrap.cc:654] T 61b81445adb34e1e934607d50aa2fcfb P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.450963 18256 tablet_bootstrap.cc:492] T 61b81445adb34e1e934607d50aa2fcfb P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 02:00:04.451170 18256 ts_tablet_manager.cc:1367] T 61b81445adb34e1e934607d50aa2fcfb P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240503 02:00:04.452342 18256 raft_consensus.cc:348] T 61b81445adb34e1e934607d50aa2fcfb P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:04.452775 18256 raft_consensus.cc:374] T 61b81445adb34e1e934607d50aa2fcfb P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.453392 18256 raft_consensus.cc:729] T 61b81445adb34e1e934607d50aa2fcfb P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.453560 18256 consensus_queue.cc:260] T 61b81445adb34e1e934607d50aa2fcfb P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:04.453888 18257 tablet_bootstrap.cc:492] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 02:00:04.455189 18257 tablet_bootstrap.cc:654] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:04.459997 18256 ts_tablet_manager.cc:1398] T 61b81445adb34e1e934607d50aa2fcfb P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20240503 02:00:04.465023 18257 tablet_bootstrap.cc:492] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 02:00:04.466149 18257 ts_tablet_manager.cc:1367] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240503 02:00:04.469981 18257 raft_consensus.cc:348] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:04.470146 18257 raft_consensus.cc:374] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:04.470227 18257 raft_consensus.cc:729] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 02:00:04.470378 18257 consensus_queue.cc:260] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:04.470858 18257 ts_tablet_manager.cc:1398] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:00:04.474660 7275 consensus_queue.cc:237] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:04.476200 7145 raft_consensus.cc:1261] T 7046186cd7f540e7b0292da5f422a4a1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:04.477192 9493 raft_consensus.cc:1261] T 7046186cd7f540e7b0292da5f422a4a1 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:04.478093 17364 consensus_queue.cc:1035] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:04.478297 17557 consensus_queue.cc:1035] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:04.481295 17364 raft_consensus.cc:2932] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:04.481614 9493 raft_consensus.cc:2932] T 7046186cd7f540e7b0292da5f422a4a1 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:04.484174 7145 raft_consensus.cc:2932] T 7046186cd7f540e7b0292da5f422a4a1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:04.487236 6736 catalog_manager.cc:5484] T 7046186cd7f540e7b0292da5f422a4a1 P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:04.491511 9605 tablet_service.cc:1514] Processing DeleteTablet for tablet 7046186cd7f540e7b0292da5f422a4a1 with delete_type TABLET_DATA_TOMBSTONED (TS b7062fbbc2b54fff9e99ec4ea5562c76 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:58562
I20240503 02:00:04.492444 18274 tablet_replica.cc:330] T 7046186cd7f540e7b0292da5f422a4a1 P b7062fbbc2b54fff9e99ec4ea5562c76: stopping tablet replica
I20240503 02:00:04.493511 18274 raft_consensus.cc:2229] T 7046186cd7f540e7b0292da5f422a4a1 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:04.493897 18274 raft_consensus.cc:2258] T 7046186cd7f540e7b0292da5f422a4a1 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:04.494740 18274 ts_tablet_manager.cc:1875] T 7046186cd7f540e7b0292da5f422a4a1 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:04.497820 18274 ts_tablet_manager.cc:1888] T 7046186cd7f540e7b0292da5f422a4a1 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:00:04.497926 18274 log.cc:1196] T 7046186cd7f540e7b0292da5f422a4a1 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/7046186cd7f540e7b0292da5f422a4a1
I20240503 02:00:04.498884 6727 catalog_manager.cc:4830] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): tablet 7046186cd7f540e7b0292da5f422a4a1 (table rebalancer_extra_table_0 [id=72ea0c008b3e43ccab2d5b224cf8896b]) successfully deleted
I20240503 02:00:04.628286 17364 raft_consensus.cc:482] T 743d87c1c6ad4f7585514f1adb350ae1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:04.628566 17364 raft_consensus.cc:504] T 743d87c1c6ad4f7585514f1adb350ae1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:04.629114 17364 leader_election.cc:288] T 743d87c1c6ad4f7585514f1adb350ae1 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 02:00:04.629861 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743d87c1c6ad4f7585514f1adb350ae1" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 02:00:04.629860 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743d87c1c6ad4f7585514f1adb350ae1" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 02:00:04.630039 7145 raft_consensus.cc:2454] T 743d87c1c6ad4f7585514f1adb350ae1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 02:00:04.630039 6882 raft_consensus.cc:2454] T 743d87c1c6ad4f7585514f1adb350ae1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 0.
I20240503 02:00:04.630462 7212 leader_election.cc:302] T 743d87c1c6ad4f7585514f1adb350ae1 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 02:00:04.630808 17364 raft_consensus.cc:2781] T 743d87c1c6ad4f7585514f1adb350ae1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:04.630868 17364 raft_consensus.cc:482] T 743d87c1c6ad4f7585514f1adb350ae1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:04.630908 17364 raft_consensus.cc:3037] T 743d87c1c6ad4f7585514f1adb350ae1 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.632292 17364 raft_consensus.cc:504] T 743d87c1c6ad4f7585514f1adb350ae1 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:04.632688 17364 leader_election.cc:288] T 743d87c1c6ad4f7585514f1adb350ae1 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Requested vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 02:00:04.633122 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743d87c1c6ad4f7585514f1adb350ae1" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 02:00:04.633256 7145 raft_consensus.cc:3037] T 743d87c1c6ad4f7585514f1adb350ae1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.633654 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743d87c1c6ad4f7585514f1adb350ae1" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 02:00:04.633780 6882 raft_consensus.cc:3037] T 743d87c1c6ad4f7585514f1adb350ae1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.634383 7145 raft_consensus.cc:2454] T 743d87c1c6ad4f7585514f1adb350ae1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 02:00:04.634918 6882 raft_consensus.cc:2454] T 743d87c1c6ad4f7585514f1adb350ae1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 02:00:04.634996 7211 leader_election.cc:302] T 743d87c1c6ad4f7585514f1adb350ae1 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 02:00:04.635205 17364 raft_consensus.cc:2781] T 743d87c1c6ad4f7585514f1adb350ae1 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:04.635643 17364 raft_consensus.cc:686] T 743d87c1c6ad4f7585514f1adb350ae1 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 02:00:04.635861 17364 consensus_queue.cc:237] T 743d87c1c6ad4f7585514f1adb350ae1 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:04.639101 6735 catalog_manager.cc:5484] T 743d87c1c6ad4f7585514f1adb350ae1 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:04.782938 17508 raft_consensus.cc:482] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:04.783181 17508 raft_consensus.cc:504] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:04.783752 17508 leader_election.cc:288] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:04.784821 9361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b81445adb34e1e934607d50aa2fcfb" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 02:00:04.784925 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b81445adb34e1e934607d50aa2fcfb" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" is_pre_election: true
I20240503 02:00:04.785115 9625 raft_consensus.cc:2454] T 61b81445adb34e1e934607d50aa2fcfb P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 02:00:04.785125 9361 raft_consensus.cc:2454] T 61b81445adb34e1e934607d50aa2fcfb P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 02:00:04.785740 6949 leader_election.cc:302] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, b7062fbbc2b54fff9e99ec4ea5562c76; no voters:
I20240503 02:00:04.786266 16636 raft_consensus.cc:2781] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:04.786381 16636 raft_consensus.cc:482] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:04.786610 16636 raft_consensus.cc:3037] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.788265 16636 raft_consensus.cc:504] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:04.788939 16636 leader_election.cc:288] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:04.789837 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b81445adb34e1e934607d50aa2fcfb" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 02:00:04.789997 9363 raft_consensus.cc:3037] T 61b81445adb34e1e934607d50aa2fcfb P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.791419 9363 raft_consensus.cc:2454] T 61b81445adb34e1e934607d50aa2fcfb P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 02:00:04.791960 6949 leader_election.cc:302] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 02:00:04.792227 16636 raft_consensus.cc:2781] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:04.792311 16636 raft_consensus.cc:686] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 02:00:04.792460 16636 consensus_queue.cc:237] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:04.794428 9625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b81445adb34e1e934607d50aa2fcfb" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76"
I20240503 02:00:04.794695 9625 raft_consensus.cc:3037] T 61b81445adb34e1e934607d50aa2fcfb P b7062fbbc2b54fff9e99ec4ea5562c76 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:04.796574 9625 raft_consensus.cc:2454] T 61b81445adb34e1e934607d50aa2fcfb P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 02:00:04.796730 6741 catalog_manager.cc:5484] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:04.829604 9342 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:04.829883 6861 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:00:04.834710 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:04.841872 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:04.851841 9736 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:04.855763 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:04.861109 9472 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:04.881613 7125 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:00:04.892421 17202 consensus_queue.cc:1035] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:04.927290 18213 consensus_queue.cc:1035] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:04.946110 6741 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:00:04.951467 9363 raft_consensus.cc:1261] T 61b81445adb34e1e934607d50aa2fcfb P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:04.952112 18260 consensus_queue.cc:1035] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:04.954511 18340 tablet.cc:1722] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:04.957185 18340 tablet.cc:1620] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef: MemRowSet was empty: no flush needed.
I20240503 02:00:04.957901 9626 raft_consensus.cc:1261] T 61b81445adb34e1e934607d50aa2fcfb P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:04.963842 6741 catalog_manager.cc:5576] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387) does not have the latest schema for tablet 61b81445adb34e1e934607d50aa2fcfb (table rebalancer_dynamic_table [id=54e17be95a014be7bd8fd4de99c8a292]). Expected version 1 got 0
I20240503 02:00:04.965371 17508 consensus_queue.cc:1035] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:04.967546 18341 tablet.cc:1722] T 61b81445adb34e1e934607d50aa2fcfb P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:04.968441 18341 tablet.cc:1620] T 61b81445adb34e1e934607d50aa2fcfb P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 02:00:04.970036 6741 catalog_manager.cc:5576] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087) does not have the latest schema for tablet 61b81445adb34e1e934607d50aa2fcfb (table rebalancer_dynamic_table [id=54e17be95a014be7bd8fd4de99c8a292]). Expected version 1 got 0
I20240503 02:00:04.970891 6736 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=54e17be95a014be7bd8fd4de99c8a292] alter complete (version 1)
I20240503 02:00:04.974365 6735 catalog_manager.cc:5576] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087) does not have the latest schema for tablet 61b81445adb34e1e934607d50aa2fcfb (table rebalancer_dynamic_table [id=54e17be95a014be7bd8fd4de99c8a292]). Expected version 1 got 0
I20240503 02:00:04.979012 18344 tablet.cc:1722] T 61b81445adb34e1e934607d50aa2fcfb P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:00:04.979755 18344 tablet.cc:1620] T 61b81445adb34e1e934607d50aa2fcfb P b7062fbbc2b54fff9e99ec4ea5562c76: MemRowSet was empty: no flush needed.
I20240503 02:00:05.005450 16785 consensus_queue.cc:1035] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:05.012063 17183 consensus_queue.cc:1035] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:05.017326 6882 consensus_queue.cc:237] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:05.020038 9363 raft_consensus.cc:1261] T f8d3d130f4a44904924f7ccf4316fc75 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:05.020993 7275 raft_consensus.cc:1261] T f8d3d130f4a44904924f7ccf4316fc75 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:05.021238 17202 consensus_queue.cc:1035] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:05.022727 17202 consensus_queue.cc:1035] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:05.024101 17202 raft_consensus.cc:2932] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:05.026499 17557 consensus_queue.cc:1035] T 743d87c1c6ad4f7585514f1adb350ae1 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:05.027343 9361 raft_consensus.cc:2932] T f8d3d130f4a44904924f7ccf4316fc75 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:05.031113 6741 catalog_manager.cc:5484] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index 2 to 3, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:05.031991 7275 raft_consensus.cc:2932] T f8d3d130f4a44904924f7ccf4316fc75 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:05.033402 17364 consensus_queue.cc:1035] T 743d87c1c6ad4f7585514f1adb350ae1 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:05.037024 9736 tablet_service.cc:1514] Processing DeleteTablet for tablet f8d3d130f4a44904924f7ccf4316fc75 with delete_type TABLET_DATA_TOMBSTONED (TS db1a0000aaa346f79c0aa4b6c5105191 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:48152
I20240503 02:00:05.037907 18369 tablet_replica.cc:330] T f8d3d130f4a44904924f7ccf4316fc75 P db1a0000aaa346f79c0aa4b6c5105191: stopping tablet replica
I20240503 02:00:05.038493 18369 raft_consensus.cc:2229] T f8d3d130f4a44904924f7ccf4316fc75 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:05.038815 18369 raft_consensus.cc:2258] T f8d3d130f4a44904924f7ccf4316fc75 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:05.040212 18369 ts_tablet_manager.cc:1875] T f8d3d130f4a44904924f7ccf4316fc75 P db1a0000aaa346f79c0aa4b6c5105191: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:05.043982 18369 ts_tablet_manager.cc:1888] T f8d3d130f4a44904924f7ccf4316fc75 P db1a0000aaa346f79c0aa4b6c5105191: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:00:05.044234 18369 log.cc:1196] T f8d3d130f4a44904924f7ccf4316fc75 P db1a0000aaa346f79c0aa4b6c5105191: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/f8d3d130f4a44904924f7ccf4316fc75
I20240503 02:00:05.045496 6726 catalog_manager.cc:4830] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): tablet f8d3d130f4a44904924f7ccf4316fc75 (table rebalancer_extra_table_42 [id=1de39c3b40b04d1483699c7fc0d0c6cb]) successfully deleted
I20240503 02:00:05.077034 6740 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:00:05.087278 18340 tablet.cc:1722] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:05.087695 18340 tablet.cc:1620] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef: MemRowSet was empty: no flush needed.
I20240503 02:00:05.090744 18341 tablet.cc:1722] T 61b81445adb34e1e934607d50aa2fcfb P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:05.091120 18341 tablet.cc:1620] T 61b81445adb34e1e934607d50aa2fcfb P 12e8c683db054df3b75c6285b98a5ad8: MemRowSet was empty: no flush needed.
I20240503 02:00:05.095086 18344 tablet.cc:1722] T 61b81445adb34e1e934607d50aa2fcfb P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:00:05.095506 18344 tablet.cc:1620] T 61b81445adb34e1e934607d50aa2fcfb P b7062fbbc2b54fff9e99ec4ea5562c76: MemRowSet was empty: no flush needed.
I20240503 02:00:05.105285 6740 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table [id=54e17be95a014be7bd8fd4de99c8a292] alter complete (version 2)
I20240503 02:00:05.177115 6738 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_extra_table_68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:00:05.193521 9736 tablet_service.cc:1467] Processing CreateTablet for tablet d1df18d990ee43a596a4656dd498b8e7 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=2d7b9439fc08472c899818be06b77ff2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:05.193926 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1df18d990ee43a596a4656dd498b8e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:05.194001 9472 tablet_service.cc:1467] Processing CreateTablet for tablet d1df18d990ee43a596a4656dd498b8e7 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=2d7b9439fc08472c899818be06b77ff2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:05.194218 9474 tablet_service.cc:1467] Processing CreateTablet for tablet beae7adf9bb3404ca03709b6ff03c7b5 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=2d7b9439fc08472c899818be06b77ff2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:05.194294 9472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1df18d990ee43a596a4656dd498b8e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:05.194449 9474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beae7adf9bb3404ca03709b6ff03c7b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:05.196897 18417 tablet_bootstrap.cc:492] T d1df18d990ee43a596a4656dd498b8e7 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 02:00:05.196976 7125 tablet_service.cc:1467] Processing CreateTablet for tablet d1df18d990ee43a596a4656dd498b8e7 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=2d7b9439fc08472c899818be06b77ff2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:05.197428 7125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1df18d990ee43a596a4656dd498b8e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:05.193637 9735 tablet_service.cc:1467] Processing CreateTablet for tablet beae7adf9bb3404ca03709b6ff03c7b5 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=2d7b9439fc08472c899818be06b77ff2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:05.197839 9735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beae7adf9bb3404ca03709b6ff03c7b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:05.197994 18417 tablet_bootstrap.cc:654] T d1df18d990ee43a596a4656dd498b8e7 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:05.198678 7124 tablet_service.cc:1467] Processing CreateTablet for tablet beae7adf9bb3404ca03709b6ff03c7b5 (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=2d7b9439fc08472c899818be06b77ff2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:05.199016 7124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beae7adf9bb3404ca03709b6ff03c7b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:05.200911 7123 tablet_service.cc:1467] Processing CreateTablet for tablet 04c3319b8d8e4872a703588d3933b32c (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=2d7b9439fc08472c899818be06b77ff2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:05.201176 7123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04c3319b8d8e4872a703588d3933b32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:05.201818 18419 tablet_bootstrap.cc:492] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 02:00:05.203997 18419 tablet_bootstrap.cc:654] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:05.204072 6992 tablet_service.cc:1467] Processing CreateTablet for tablet 04c3319b8d8e4872a703588d3933b32c (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=2d7b9439fc08472c899818be06b77ff2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:05.204384 6992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04c3319b8d8e4872a703588d3933b32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:05.205322 18419 tablet_bootstrap.cc:492] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 02:00:05.205417 18419 ts_tablet_manager.cc:1367] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:00:05.205927 18419 raft_consensus.cc:348] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:05.206032 18419 raft_consensus.cc:374] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:05.206060 18419 raft_consensus.cc:729] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 02:00:05.206163 18419 consensus_queue.cc:260] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:05.206496 18419 ts_tablet_manager.cc:1398] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:05.206992 18419 tablet_bootstrap.cc:492] T d1df18d990ee43a596a4656dd498b8e7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 02:00:05.208135 18419 tablet_bootstrap.cc:654] T d1df18d990ee43a596a4656dd498b8e7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:05.211071 6739 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:00:05.213021 9474 tablet_service.cc:1467] Processing CreateTablet for tablet 04c3319b8d8e4872a703588d3933b32c (DEFAULT_TABLE table=rebalancer_extra_table_68 [id=2d7b9439fc08472c899818be06b77ff2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:00:05.213362 9474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04c3319b8d8e4872a703588d3933b32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:05.223259 18419 tablet_bootstrap.cc:492] T d1df18d990ee43a596a4656dd498b8e7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 02:00:05.223441 18419 ts_tablet_manager.cc:1367] T d1df18d990ee43a596a4656dd498b8e7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.001s
I20240503 02:00:05.223734 18340 tablet.cc:1722] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:05.223999 18419 raft_consensus.cc:348] T d1df18d990ee43a596a4656dd498b8e7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:05.224187 18419 raft_consensus.cc:374] T d1df18d990ee43a596a4656dd498b8e7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:05.224253 18419 raft_consensus.cc:729] T d1df18d990ee43a596a4656dd498b8e7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 02:00:05.224418 18419 consensus_queue.cc:260] T d1df18d990ee43a596a4656dd498b8e7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:05.224774 18419 ts_tablet_manager.cc:1398] T d1df18d990ee43a596a4656dd498b8e7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:05.228686 18344 tablet.cc:1722] T 61b81445adb34e1e934607d50aa2fcfb P b7062fbbc2b54fff9e99ec4ea5562c76: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:05.229139 18419 tablet_bootstrap.cc:492] T 04c3319b8d8e4872a703588d3933b32c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 02:00:05.229755 18341 tablet.cc:1722] T 61b81445adb34e1e934607d50aa2fcfb P 12e8c683db054df3b75c6285b98a5ad8: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:00:05.230321 18419 tablet_bootstrap.cc:654] T 04c3319b8d8e4872a703588d3933b32c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:05.232322 18418 tablet_bootstrap.cc:492] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 02:00:05.232769 18424 tablet_bootstrap.cc:492] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 02:00:05.233433 18418 tablet_bootstrap.cc:654] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:05.234840 18418 tablet_bootstrap.cc:492] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 02:00:05.234934 18418 ts_tablet_manager.cc:1367] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:00:05.235466 18424 tablet_bootstrap.cc:654] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:05.235474 18418 raft_consensus.cc:348] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:05.235595 18418 raft_consensus.cc:374] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:05.235849 18418 raft_consensus.cc:729] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 02:00:05.236140 18418 consensus_queue.cc:260] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:05.239018 18417 tablet_bootstrap.cc:492] T d1df18d990ee43a596a4656dd498b8e7 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 02:00:05.239147 18417 ts_tablet_manager.cc:1367] T d1df18d990ee43a596a4656dd498b8e7 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.000s
I20240503 02:00:05.239193 18424 tablet_bootstrap.cc:492] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef: No bootstrap required, opened a new log
I20240503 02:00:05.239301 18424 ts_tablet_manager.cc:1367] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:00:05.236337 18419 tablet_bootstrap.cc:492] T 04c3319b8d8e4872a703588d3933b32c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: No bootstrap required, opened a new log
I20240503 02:00:05.239692 18419 ts_tablet_manager.cc:1367] T 04c3319b8d8e4872a703588d3933b32c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:00:05.239823 18424 raft_consensus.cc:348] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:05.239943 18424 raft_consensus.cc:374] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:05.239975 18424 raft_consensus.cc:729] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 02:00:05.240114 18424 consensus_queue.cc:260] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:05.240278 6741 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: rebalancer_dynamic_table_new_name [id=54e17be95a014be7bd8fd4de99c8a292] alter complete (version 3)
I20240503 02:00:05.240437 18424 ts_tablet_manager.cc:1398] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:05.240461 18419 raft_consensus.cc:348] T 04c3319b8d8e4872a703588d3933b32c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:05.240561 18419 raft_consensus.cc:374] T 04c3319b8d8e4872a703588d3933b32c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:05.240610 18419 raft_consensus.cc:729] T 04c3319b8d8e4872a703588d3933b32c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 02:00:05.240790 18419 consensus_queue.cc:260] T 04c3319b8d8e4872a703588d3933b32c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:05.242280 18419 ts_tablet_manager.cc:1398] T 04c3319b8d8e4872a703588d3933b32c P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:00:05.242576 18418 ts_tablet_manager.cc:1398] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:00:05.242784 18418 tablet_bootstrap.cc:492] T beae7adf9bb3404ca03709b6ff03c7b5 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 02:00:05.244837 18418 tablet_bootstrap.cc:654] T beae7adf9bb3404ca03709b6ff03c7b5 P db1a0000aaa346f79c0aa4b6c5105191: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:05.239671 18417 raft_consensus.cc:348] T d1df18d990ee43a596a4656dd498b8e7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:05.254052 18417 raft_consensus.cc:374] T d1df18d990ee43a596a4656dd498b8e7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:05.254153 18417 raft_consensus.cc:729] T d1df18d990ee43a596a4656dd498b8e7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 02:00:05.255422 18417 consensus_queue.cc:260] T d1df18d990ee43a596a4656dd498b8e7 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:05.255865 18417 ts_tablet_manager.cc:1398] T d1df18d990ee43a596a4656dd498b8e7 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.017s user 0.000s sys 0.002s
I20240503 02:00:05.256875 18417 tablet_bootstrap.cc:492] T beae7adf9bb3404ca03709b6ff03c7b5 P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 02:00:05.262250 18417 tablet_bootstrap.cc:654] T beae7adf9bb3404ca03709b6ff03c7b5 P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:05.264734 18418 tablet_bootstrap.cc:492] T beae7adf9bb3404ca03709b6ff03c7b5 P db1a0000aaa346f79c0aa4b6c5105191: No bootstrap required, opened a new log
I20240503 02:00:05.264986 18418 ts_tablet_manager.cc:1367] T beae7adf9bb3404ca03709b6ff03c7b5 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20240503 02:00:05.265379 18417 tablet_bootstrap.cc:492] T beae7adf9bb3404ca03709b6ff03c7b5 P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 02:00:05.265530 18417 ts_tablet_manager.cc:1367] T beae7adf9bb3404ca03709b6ff03c7b5 P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20240503 02:00:05.265609 18418 raft_consensus.cc:348] T beae7adf9bb3404ca03709b6ff03c7b5 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:05.265794 18418 raft_consensus.cc:374] T beae7adf9bb3404ca03709b6ff03c7b5 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:05.265872 18418 raft_consensus.cc:729] T beae7adf9bb3404ca03709b6ff03c7b5 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 02:00:05.266160 18418 consensus_queue.cc:260] T beae7adf9bb3404ca03709b6ff03c7b5 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:05.266156 18417 raft_consensus.cc:348] T beae7adf9bb3404ca03709b6ff03c7b5 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:05.266261 18417 raft_consensus.cc:374] T beae7adf9bb3404ca03709b6ff03c7b5 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:05.266319 18417 raft_consensus.cc:729] T beae7adf9bb3404ca03709b6ff03c7b5 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 02:00:05.266487 18417 consensus_queue.cc:260] T beae7adf9bb3404ca03709b6ff03c7b5 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:05.266948 18417 ts_tablet_manager.cc:1398] T beae7adf9bb3404ca03709b6ff03c7b5 P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:05.267334 18417 tablet_bootstrap.cc:492] T 04c3319b8d8e4872a703588d3933b32c P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 02:00:05.268922 16636 raft_consensus.cc:482] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:05.269038 16636 raft_consensus.cc:504] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:05.269340 18418 ts_tablet_manager.cc:1398] T beae7adf9bb3404ca03709b6ff03c7b5 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:00:05.269445 16636 leader_election.cc:288] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 02:00:05.270184 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04c3319b8d8e4872a703588d3933b32c" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 02:00:05.270341 7143 raft_consensus.cc:2454] T 04c3319b8d8e4872a703588d3933b32c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 0.
I20240503 02:00:05.270753 6949 leader_election.cc:302] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 02:00:05.271237 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04c3319b8d8e4872a703588d3933b32c" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 02:00:05.271795 17508 raft_consensus.cc:2781] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:05.271855 17508 raft_consensus.cc:482] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:05.271893 17508 raft_consensus.cc:3037] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:00:05.272051 6947 leader_election.cc:341] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:00:05.273204 17508 raft_consensus.cc:504] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:05.273754 17508 leader_election.cc:288] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Requested vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 02:00:05.273770 18417 tablet_bootstrap.cc:654] T 04c3319b8d8e4872a703588d3933b32c P bb5364d730cb4489a0bfc6c270b7450a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:05.274446 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04c3319b8d8e4872a703588d3933b32c" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
W20240503 02:00:05.274878 6947 leader_election.cc:341] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:00:05.274857 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04c3319b8d8e4872a703588d3933b32c" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 02:00:05.274979 7143 raft_consensus.cc:3037] T 04c3319b8d8e4872a703588d3933b32c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:05.276160 7143 raft_consensus.cc:2454] T 04c3319b8d8e4872a703588d3933b32c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 02:00:05.277266 6949 leader_election.cc:302] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:05.277776 17508 raft_consensus.cc:2781] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:05.278008 17508 raft_consensus.cc:686] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 02:00:05.278219 17508 consensus_queue.cc:237] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:05.282335 6739 catalog_manager.cc:5484] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:05.286695 18417 tablet_bootstrap.cc:492] T 04c3319b8d8e4872a703588d3933b32c P bb5364d730cb4489a0bfc6c270b7450a: No bootstrap required, opened a new log
I20240503 02:00:05.286918 18417 ts_tablet_manager.cc:1367] T 04c3319b8d8e4872a703588d3933b32c P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.002s
I20240503 02:00:05.288122 18417 raft_consensus.cc:348] T 04c3319b8d8e4872a703588d3933b32c P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:05.288369 18417 raft_consensus.cc:374] T 04c3319b8d8e4872a703588d3933b32c P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:05.288470 18417 raft_consensus.cc:729] T 04c3319b8d8e4872a703588d3933b32c P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 02:00:05.288712 18417 consensus_queue.cc:260] T 04c3319b8d8e4872a703588d3933b32c P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:05.289180 18417 ts_tablet_manager.cc:1398] T 04c3319b8d8e4872a703588d3933b32c P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:05.340962 6992 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 02:00:05.355075 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:05.362600 6861 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:00:05.363137 6740 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:05.363294 6740 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41614:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:00:05.365921 7123 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:00:05.366868 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:05.371877 6740 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 54aa976bc02b497e8f9115c0d4679ef6: Sending DeleteTablet for 3 replicas of tablet 61b81445adb34e1e934607d50aa2fcfb
I20240503 02:00:05.375680 9342 tablet_service.cc:1514] Processing DeleteTablet for tablet 61b81445adb34e1e934607d50aa2fcfb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:05 UTC) from {username='slave'} at 127.0.0.1:36040
I20240503 02:00:05.376607 6992 tablet_service.cc:1514] Processing DeleteTablet for tablet 61b81445adb34e1e934607d50aa2fcfb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:05 UTC) from {username='slave'} at 127.0.0.1:35240
I20240503 02:00:05.378716 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:05.380472 9605 tablet_service.cc:1514] Processing DeleteTablet for tablet 61b81445adb34e1e934607d50aa2fcfb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:00:05 UTC) from {username='slave'} at 127.0.0.1:58562
I20240503 02:00:05.383328 9474 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:05.403843 17532 raft_consensus.cc:482] T d1df18d990ee43a596a4656dd498b8e7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:05.404042 17532 raft_consensus.cc:504] T d1df18d990ee43a596a4656dd498b8e7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:05.404749 17532 leader_election.cc:288] T d1df18d990ee43a596a4656dd498b8e7 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 02:00:05.405339 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1df18d990ee43a596a4656dd498b8e7" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 02:00:05.405638 7145 raft_consensus.cc:2454] T d1df18d990ee43a596a4656dd498b8e7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 02:00:05.406148 9429 leader_election.cc:302] T d1df18d990ee43a596a4656dd498b8e7 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 02:00:05.406608 17532 raft_consensus.cc:2781] T d1df18d990ee43a596a4656dd498b8e7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:05.406702 17532 raft_consensus.cc:482] T d1df18d990ee43a596a4656dd498b8e7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:05.406749 17532 raft_consensus.cc:3037] T d1df18d990ee43a596a4656dd498b8e7 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:05.408200 17532 raft_consensus.cc:504] T d1df18d990ee43a596a4656dd498b8e7 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:05.408599 17532 leader_election.cc:288] T d1df18d990ee43a596a4656dd498b8e7 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Requested vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 02:00:05.409025 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1df18d990ee43a596a4656dd498b8e7" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 02:00:05.409142 7145 raft_consensus.cc:3037] T d1df18d990ee43a596a4656dd498b8e7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:05.410957 7145 raft_consensus.cc:2454] T d1df18d990ee43a596a4656dd498b8e7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 02:00:05.411352 9429 leader_election.cc:302] T d1df18d990ee43a596a4656dd498b8e7 P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 02:00:05.411579 17532 raft_consensus.cc:2781] T d1df18d990ee43a596a4656dd498b8e7 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:05.411746 17532 raft_consensus.cc:686] T d1df18d990ee43a596a4656dd498b8e7 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Becoming Leader. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Running, Role: LEADER
I20240503 02:00:05.411880 17532 consensus_queue.cc:237] T d1df18d990ee43a596a4656dd498b8e7 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:05.412092 9755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1df18d990ee43a596a4656dd498b8e7" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 02:00:05.412225 9755 raft_consensus.cc:2454] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 0.
I20240503 02:00:05.412106 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1df18d990ee43a596a4656dd498b8e7" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 02:00:05.412432 9756 raft_consensus.cc:3037] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:05.413897 9756 raft_consensus.cc:2454] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5364d730cb4489a0bfc6c270b7450a in term 1.
I20240503 02:00:05.414346 18446 tablet_replica.cc:330] T 61b81445adb34e1e934607d50aa2fcfb P b7062fbbc2b54fff9e99ec4ea5562c76: stopping tablet replica
I20240503 02:00:05.416375 18446 raft_consensus.cc:2229] T 61b81445adb34e1e934607d50aa2fcfb P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:05.416563 18446 raft_consensus.cc:2258] T 61b81445adb34e1e934607d50aa2fcfb P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:05.423431 6740 catalog_manager.cc:5484] T d1df18d990ee43a596a4656dd498b8e7 P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: term changed from 0 to 1, leader changed from <none> to bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134). New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:05.427724 18444 tablet_replica.cc:330] T 61b81445adb34e1e934607d50aa2fcfb P 12e8c683db054df3b75c6285b98a5ad8: stopping tablet replica
I20240503 02:00:05.429196 18444 raft_consensus.cc:2229] T 61b81445adb34e1e934607d50aa2fcfb P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:05.429917 18446 ts_tablet_manager.cc:1875] T 61b81445adb34e1e934607d50aa2fcfb P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:05.431030 18444 raft_consensus.cc:2258] T 61b81445adb34e1e934607d50aa2fcfb P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:05.431290 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:05.434013 18444 ts_tablet_manager.cc:1875] T 61b81445adb34e1e934607d50aa2fcfb P 12e8c683db054df3b75c6285b98a5ad8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:05.441581 18444 ts_tablet_manager.cc:1888] T 61b81445adb34e1e934607d50aa2fcfb P 12e8c683db054df3b75c6285b98a5ad8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:05.441807 18444 log.cc:1196] T 61b81445adb34e1e934607d50aa2fcfb P 12e8c683db054df3b75c6285b98a5ad8: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/61b81445adb34e1e934607d50aa2fcfb
I20240503 02:00:05.442209 18444 ts_tablet_manager.cc:1909] T 61b81445adb34e1e934607d50aa2fcfb P 12e8c683db054df3b75c6285b98a5ad8: Deleting consensus metadata
I20240503 02:00:05.446173 6727 catalog_manager.cc:4830] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): tablet 61b81445adb34e1e934607d50aa2fcfb (table rebalancer_dynamic_table_new_name [id=54e17be95a014be7bd8fd4de99c8a292]) successfully deleted
I20240503 02:00:05.449318 18115 raft_consensus.cc:482] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:05.449438 18115 raft_consensus.cc:504] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:05.449827 18115 leader_election.cc:288] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 02:00:05.450939 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beae7adf9bb3404ca03709b6ff03c7b5" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a" is_pre_election: true
I20240503 02:00:05.451036 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beae7adf9bb3404ca03709b6ff03c7b5" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 02:00:05.451097 9494 raft_consensus.cc:2454] T beae7adf9bb3404ca03709b6ff03c7b5 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 02:00:05.451201 9756 raft_consensus.cc:2454] T beae7adf9bb3404ca03709b6ff03c7b5 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 0.
I20240503 02:00:05.451678 7079 leader_election.cc:302] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 02:00:05.452072 16780 raft_consensus.cc:2781] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:05.452126 16780 raft_consensus.cc:482] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:05.452158 16780 raft_consensus.cc:3037] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:05.453341 16780 raft_consensus.cc:504] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:05.453727 16780 leader_election.cc:288] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Requested vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 02:00:05.454320 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beae7adf9bb3404ca03709b6ff03c7b5" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 02:00:05.454453 9756 raft_consensus.cc:3037] T beae7adf9bb3404ca03709b6ff03c7b5 P db1a0000aaa346f79c0aa4b6c5105191 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:05.454766 9494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beae7adf9bb3404ca03709b6ff03c7b5" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 02:00:05.454905 9494 raft_consensus.cc:3037] T beae7adf9bb3404ca03709b6ff03c7b5 P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:05.455648 9756 raft_consensus.cc:2454] T beae7adf9bb3404ca03709b6ff03c7b5 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 02:00:05.455930 18445 tablet_replica.cc:330] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 02:00:05.455960 7079 leader_election.cc:302] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 02:00:05.455986 9494 raft_consensus.cc:2454] T beae7adf9bb3404ca03709b6ff03c7b5 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 02:00:05.456131 18445 raft_consensus.cc:2229] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:00:05.456344 18115 raft_consensus.cc:2781] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:05.456398 18445 raft_consensus.cc:2258] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:05.456429 18115 raft_consensus.cc:686] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Becoming Leader. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Running, Role: LEADER
I20240503 02:00:05.456555 18115 consensus_queue.cc:237] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:05.458298 18445 ts_tablet_manager.cc:1875] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:00:05.459681 6740 catalog_manager.cc:5484] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131). New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:05.462606 18446 ts_tablet_manager.cc:1888] T 61b81445adb34e1e934607d50aa2fcfb P b7062fbbc2b54fff9e99ec4ea5562c76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:05.462692 18446 log.cc:1196] T 61b81445adb34e1e934607d50aa2fcfb P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/61b81445adb34e1e934607d50aa2fcfb
I20240503 02:00:05.462981 18446 ts_tablet_manager.cc:1909] T 61b81445adb34e1e934607d50aa2fcfb P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting consensus metadata
I20240503 02:00:05.467325 6727 catalog_manager.cc:4830] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): tablet 61b81445adb34e1e934607d50aa2fcfb (table rebalancer_dynamic_table_new_name [id=54e17be95a014be7bd8fd4de99c8a292]) successfully deleted
I20240503 02:00:05.468508 18445 ts_tablet_manager.cc:1888] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:00:05.468672 18445 log.cc:1196] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/61b81445adb34e1e934607d50aa2fcfb
I20240503 02:00:05.469431 18445 ts_tablet_manager.cc:1909] T 61b81445adb34e1e934607d50aa2fcfb P 63331f455be049c1bdd6c23c59ac1bef: Deleting consensus metadata
I20240503 02:00:05.470227 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet 61b81445adb34e1e934607d50aa2fcfb (table rebalancer_dynamic_table_new_name [id=54e17be95a014be7bd8fd4de99c8a292]) successfully deleted
I20240503 02:00:05.474947 6740 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:00:05.482604 6861 tablet_service.cc:1467] Processing CreateTablet for tablet 27ac67e7e677405e89c22d22f1e00f93 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a36762e13a5b48418257350c01974c5b]), partition=
I20240503 02:00:05.482932 6861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27ac67e7e677405e89c22d22f1e00f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:05.487299 9341 tablet_service.cc:1467] Processing CreateTablet for tablet 27ac67e7e677405e89c22d22f1e00f93 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a36762e13a5b48418257350c01974c5b]), partition=
I20240503 02:00:05.487740 7254 tablet_service.cc:1467] Processing CreateTablet for tablet 27ac67e7e677405e89c22d22f1e00f93 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a36762e13a5b48418257350c01974c5b]), partition=
I20240503 02:00:05.487758 9341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27ac67e7e677405e89c22d22f1e00f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:05.488000 7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27ac67e7e677405e89c22d22f1e00f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:05.493289 18452 tablet_bootstrap.cc:492] T 27ac67e7e677405e89c22d22f1e00f93 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 02:00:05.493587 17532 raft_consensus.cc:482] T 04c3319b8d8e4872a703588d3933b32c P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:05.493757 17532 raft_consensus.cc:504] T 04c3319b8d8e4872a703588d3933b32c P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:05.494297 17532 leader_election.cc:288] T 04c3319b8d8e4872a703588d3933b32c P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 02:00:05.495095 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04c3319b8d8e4872a703588d3933b32c" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 02:00:05.495235 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04c3319b8d8e4872a703588d3933b32c" candidate_uuid: "bb5364d730cb4489a0bfc6c270b7450a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 02:00:05.495395 7143 raft_consensus.cc:2379] T 04c3319b8d8e4872a703588d3933b32c P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bb5364d730cb4489a0bfc6c270b7450a in current term 1: Already voted for candidate 63331f455be049c1bdd6c23c59ac1bef in this term.
I20240503 02:00:05.495406 18451 tablet_bootstrap.cc:492] T 27ac67e7e677405e89c22d22f1e00f93 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 02:00:05.495745 9429 leader_election.cc:302] T 04c3319b8d8e4872a703588d3933b32c P bb5364d730cb4489a0bfc6c270b7450a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb5364d730cb4489a0bfc6c270b7450a; no voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a
I20240503 02:00:05.495963 17532 raft_consensus.cc:3037] T 04c3319b8d8e4872a703588d3933b32c P bb5364d730cb4489a0bfc6c270b7450a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:05.496860 18450 tablet_bootstrap.cc:492] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 02:00:05.497150 17532 raft_consensus.cc:2726] T 04c3319b8d8e4872a703588d3933b32c P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:00:05.498347 18450 tablet_bootstrap.cc:654] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:05.499166 18451 tablet_bootstrap.cc:654] T 27ac67e7e677405e89c22d22f1e00f93 P 1d0e738c34af40fa9950c6f086619b3a: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:05.500633 18451 tablet_bootstrap.cc:492] T 27ac67e7e677405e89c22d22f1e00f93 P 1d0e738c34af40fa9950c6f086619b3a: No bootstrap required, opened a new log
I20240503 02:00:05.500813 18451 ts_tablet_manager.cc:1367] T 27ac67e7e677405e89c22d22f1e00f93 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:00:05.501962 18451 raft_consensus.cc:348] T 27ac67e7e677405e89c22d22f1e00f93 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:05.502180 18451 raft_consensus.cc:374] T 27ac67e7e677405e89c22d22f1e00f93 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:05.502214 18452 tablet_bootstrap.cc:654] T 27ac67e7e677405e89c22d22f1e00f93 P 12e8c683db054df3b75c6285b98a5ad8: Neither blocks nor log segments found. Creating new log.
I20240503 02:00:05.502272 18451 raft_consensus.cc:729] T 27ac67e7e677405e89c22d22f1e00f93 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 02:00:05.502452 18451 consensus_queue.cc:260] T 27ac67e7e677405e89c22d22f1e00f93 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:05.504623 18451 ts_tablet_manager.cc:1398] T 27ac67e7e677405e89c22d22f1e00f93 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:00:05.508893 18450 tablet_bootstrap.cc:492] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46: No bootstrap required, opened a new log
I20240503 02:00:05.509006 18450 ts_tablet_manager.cc:1367] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:00:05.509552 18450 raft_consensus.cc:348] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:05.509778 18450 raft_consensus.cc:374] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:05.509864 18450 raft_consensus.cc:729] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 02:00:05.510082 18450 consensus_queue.cc:260] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:05.510970 18450 ts_tablet_manager.cc:1398] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:05.519237 18452 tablet_bootstrap.cc:492] T 27ac67e7e677405e89c22d22f1e00f93 P 12e8c683db054df3b75c6285b98a5ad8: No bootstrap required, opened a new log
I20240503 02:00:05.519357 18452 ts_tablet_manager.cc:1367] T 27ac67e7e677405e89c22d22f1e00f93 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.002s
I20240503 02:00:05.519896 18452 raft_consensus.cc:348] T 27ac67e7e677405e89c22d22f1e00f93 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:05.520012 18452 raft_consensus.cc:374] T 27ac67e7e677405e89c22d22f1e00f93 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:00:05.520040 18452 raft_consensus.cc:729] T 27ac67e7e677405e89c22d22f1e00f93 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 02:00:05.520162 18452 consensus_queue.cc:260] T 27ac67e7e677405e89c22d22f1e00f93 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:05.520380 18452 ts_tablet_manager.cc:1398] T 27ac67e7e677405e89c22d22f1e00f93 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:05.542039 9755 consensus_queue.cc:237] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:05.543361 9494 raft_consensus.cc:1261] T b3ce9de18bda410aa204bf7c1c32e1b7 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:05.543391 6882 raft_consensus.cc:1261] T b3ce9de18bda410aa204bf7c1c32e1b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:05.543897 16785 consensus_queue.cc:1035] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:05.544634 16785 consensus_queue.cc:1035] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:05.547367 16785 raft_consensus.cc:2932] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:05.549582 6882 raft_consensus.cc:2932] T b3ce9de18bda410aa204bf7c1c32e1b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:05.550302 9494 raft_consensus.cc:2932] T b3ce9de18bda410aa204bf7c1c32e1b7 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:05.551234 6738 catalog_manager.cc:5484] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: config changed from index 2 to 3, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:05.559228 6992 tablet_service.cc:1514] Processing DeleteTablet for tablet b3ce9de18bda410aa204bf7c1c32e1b7 with delete_type TABLET_DATA_TOMBSTONED (TS 63331f455be049c1bdd6c23c59ac1bef not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:35240
I20240503 02:00:05.559661 18445 tablet_replica.cc:330] T b3ce9de18bda410aa204bf7c1c32e1b7 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 02:00:05.559880 18445 raft_consensus.cc:2229] T b3ce9de18bda410aa204bf7c1c32e1b7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:05.560025 18445 raft_consensus.cc:2258] T b3ce9de18bda410aa204bf7c1c32e1b7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:05.560493 18445 ts_tablet_manager.cc:1875] T b3ce9de18bda410aa204bf7c1c32e1b7 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:05.564258 18445 ts_tablet_manager.cc:1888] T b3ce9de18bda410aa204bf7c1c32e1b7 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:00:05.564358 18445 log.cc:1196] T b3ce9de18bda410aa204bf7c1c32e1b7 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/b3ce9de18bda410aa204bf7c1c32e1b7
I20240503 02:00:05.565052 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet b3ce9de18bda410aa204bf7c1c32e1b7 (table rebalancer_extra_table_14 [id=607fa4fc037b4c5ba04342cb6d7fd017]) successfully deleted
I20240503 02:00:05.650755 18332 raft_consensus.cc:482] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:05.651008 18332 raft_consensus.cc:504] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:05.651611 18332 leader_election.cc:288] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:05.652045 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ac67e7e677405e89c22d22f1e00f93" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8" is_pre_election: true
I20240503 02:00:05.652225 9363 raft_consensus.cc:2454] T 27ac67e7e677405e89c22d22f1e00f93 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 02:00:05.652266 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ac67e7e677405e89c22d22f1e00f93" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 02:00:05.652442 7275 raft_consensus.cc:2454] T 27ac67e7e677405e89c22d22f1e00f93 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 0.
I20240503 02:00:05.652619 6818 leader_election.cc:302] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 02:00:05.652948 18332 raft_consensus.cc:2781] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:00:05.653018 18332 raft_consensus.cc:482] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:00:05.653070 18332 raft_consensus.cc:3037] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:05.654430 18332 raft_consensus.cc:504] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:05.654933 18332 leader_election.cc:288] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:05.655709 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ac67e7e677405e89c22d22f1e00f93" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 02:00:05.655848 7275 raft_consensus.cc:3037] T 27ac67e7e677405e89c22d22f1e00f93 P 1d0e738c34af40fa9950c6f086619b3a [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:05.656993 7275 raft_consensus.cc:2454] T 27ac67e7e677405e89c22d22f1e00f93 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 02:00:05.658591 6817 leader_election.cc:302] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 02:00:05.658926 9363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ac67e7e677405e89c22d22f1e00f93" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 02:00:05.659061 9363 raft_consensus.cc:3037] T 27ac67e7e677405e89c22d22f1e00f93 P 12e8c683db054df3b75c6285b98a5ad8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:00:05.660327 9363 raft_consensus.cc:2454] T 27ac67e7e677405e89c22d22f1e00f93 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 02:00:05.662011 17202 raft_consensus.cc:2781] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:00:05.662235 17202 raft_consensus.cc:686] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 02:00:05.662411 17202 consensus_queue.cc:237] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:05.667863 6738 catalog_manager.cc:5484] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:05.715775 16636 consensus_queue.cc:1035] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:05.719818 16636 consensus_queue.cc:1035] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:05.744333 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:05.745196 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:05.755311 6861 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:00:05.765237 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:05.771310 9474 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:00:05.777441 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:05.790400 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:05.804276 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:05.885138 7143 consensus_queue.cc:237] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:05.886912 7012 raft_consensus.cc:1261] T d80398c9a264430197262bb42241d58d P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:05.886915 9493 raft_consensus.cc:1261] T d80398c9a264430197262bb42241d58d P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:05.887327 18115 consensus_queue.cc:1035] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:05.887709 16780 consensus_queue.cc:1035] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:05.889425 18115 raft_consensus.cc:2932] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 02:00:05.889853 9493 raft_consensus.cc:2932] T d80398c9a264430197262bb42241d58d P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 02:00:05.890695 7012 raft_consensus.cc:2932] T d80398c9a264430197262bb42241d58d P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 02:00:05.894013 6741 catalog_manager.cc:5484] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 02:00:05.897464 9605 tablet_service.cc:1514] Processing DeleteTablet for tablet d80398c9a264430197262bb42241d58d with delete_type TABLET_DATA_TOMBSTONED (TS b7062fbbc2b54fff9e99ec4ea5562c76 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:58562
I20240503 02:00:05.897733 18446 tablet_replica.cc:330] T d80398c9a264430197262bb42241d58d P b7062fbbc2b54fff9e99ec4ea5562c76: stopping tablet replica
I20240503 02:00:05.897879 18446 raft_consensus.cc:2229] T d80398c9a264430197262bb42241d58d P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:05.898020 18446 raft_consensus.cc:2258] T d80398c9a264430197262bb42241d58d P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:05.898452 18446 ts_tablet_manager.cc:1875] T d80398c9a264430197262bb42241d58d P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:05.900801 18446 ts_tablet_manager.cc:1888] T d80398c9a264430197262bb42241d58d P b7062fbbc2b54fff9e99ec4ea5562c76: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:00:05.900882 18446 log.cc:1196] T d80398c9a264430197262bb42241d58d P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/d80398c9a264430197262bb42241d58d
I20240503 02:00:05.902032 6727 catalog_manager.cc:4830] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): tablet d80398c9a264430197262bb42241d58d (table rebalancer_extra_table_42 [id=1de39c3b40b04d1483699c7fc0d0c6cb]) successfully deleted
I20240503 02:00:05.902997 16780 consensus_queue.cc:1035] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:05.907158 18115 consensus_queue.cc:1035] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:05.935742 16828 consensus_queue.cc:1035] T d1df18d990ee43a596a4656dd498b8e7 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:05.939224 16828 consensus_queue.cc:1035] T d1df18d990ee43a596a4656dd498b8e7 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:06.091334 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:06.092314 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:06.092516 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:06.119949 18517 consensus_queue.cc:1035] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:06.120625 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:06.140599 6861 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:00:06.144251 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:06.154671 18332 consensus_queue.cc:1035] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:00:06.164795 9474 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:00:06.165483 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:06.266181 7143 consensus_queue.cc:237] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:06.268088 9756 raft_consensus.cc:1261] T 76e8697ac8624e03a46bddae35f4c9fa P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:06.268185 9493 raft_consensus.cc:1261] T 76e8697ac8624e03a46bddae35f4c9fa P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:06.268602 18554 consensus_queue.cc:1035] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:06.269275 18554 consensus_queue.cc:1035] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:06.271842 18115 raft_consensus.cc:2932] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:06.272805 9757 raft_consensus.cc:2932] T 76e8697ac8624e03a46bddae35f4c9fa P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:06.277086 6733 catalog_manager.cc:5484] T 76e8697ac8624e03a46bddae35f4c9fa P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: config changed from index 6 to 7, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:06.279966 9493 raft_consensus.cc:2932] T 76e8697ac8624e03a46bddae35f4c9fa P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:06.281728 7254 tablet_service.cc:1514] Processing DeleteTablet for tablet 76e8697ac8624e03a46bddae35f4c9fa with delete_type TABLET_DATA_TOMBSTONED (TS 1d0e738c34af40fa9950c6f086619b3a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45608
I20240503 02:00:06.282104 18640 tablet_replica.cc:330] T 76e8697ac8624e03a46bddae35f4c9fa P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 02:00:06.282306 18640 raft_consensus.cc:2229] T 76e8697ac8624e03a46bddae35f4c9fa P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:06.282573 18640 raft_consensus.cc:2258] T 76e8697ac8624e03a46bddae35f4c9fa P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:06.283478 18640 ts_tablet_manager.cc:1875] T 76e8697ac8624e03a46bddae35f4c9fa P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:06.286557 18640 ts_tablet_manager.cc:1888] T 76e8697ac8624e03a46bddae35f4c9fa P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:00:06.286747 18640 log.cc:1196] T 76e8697ac8624e03a46bddae35f4c9fa P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/76e8697ac8624e03a46bddae35f4c9fa
I20240503 02:00:06.287446 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet 76e8697ac8624e03a46bddae35f4c9fa (table rebalance_test_table_7 [id=3428bac98e0c49908450b3b51cb9d18c]) successfully deleted
I20240503 02:00:06.469579 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:06.470360 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:06.477273 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:06.482461 9474 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:00:06.488912 6861 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:00:06.489821 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:06.493906 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:06.499326 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:06.614418 9757 consensus_queue.cc:237] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:06.615849 6881 raft_consensus.cc:1261] T 9e21d082e40540d4bb908fc9e90055c9 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:06.615839 7143 raft_consensus.cc:1261] T 9e21d082e40540d4bb908fc9e90055c9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:06.616263 18474 consensus_queue.cc:1035] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:06.616457 17183 consensus_queue.cc:1035] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:06.619395 18474 raft_consensus.cc:2932] T 9e21d082e40540d4bb908fc9e90055c9 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:06.620352 6881 raft_consensus.cc:2932] T 9e21d082e40540d4bb908fc9e90055c9 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:06.623591 7143 raft_consensus.cc:2932] T 9e21d082e40540d4bb908fc9e90055c9 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:06.625593 6737 catalog_manager.cc:5484] T 9e21d082e40540d4bb908fc9e90055c9 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:06.630277 9341 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e21d082e40540d4bb908fc9e90055c9 with delete_type TABLET_DATA_TOMBSTONED (TS 12e8c683db054df3b75c6285b98a5ad8 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:36040
I20240503 02:00:06.632100 18713 tablet_replica.cc:330] T 9e21d082e40540d4bb908fc9e90055c9 P 12e8c683db054df3b75c6285b98a5ad8: stopping tablet replica
I20240503 02:00:06.632346 18713 raft_consensus.cc:2229] T 9e21d082e40540d4bb908fc9e90055c9 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:06.632566 18713 raft_consensus.cc:2258] T 9e21d082e40540d4bb908fc9e90055c9 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:06.633057 18713 ts_tablet_manager.cc:1875] T 9e21d082e40540d4bb908fc9e90055c9 P 12e8c683db054df3b75c6285b98a5ad8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:06.642288 18713 ts_tablet_manager.cc:1888] T 9e21d082e40540d4bb908fc9e90055c9 P 12e8c683db054df3b75c6285b98a5ad8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:00:06.642439 18713 log.cc:1196] T 9e21d082e40540d4bb908fc9e90055c9 P 12e8c683db054df3b75c6285b98a5ad8: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/9e21d082e40540d4bb908fc9e90055c9
I20240503 02:00:06.643374 6727 catalog_manager.cc:4830] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): tablet 9e21d082e40540d4bb908fc9e90055c9 (table rebalancer_extra_table_18 [id=9c9f7f12e036416d8388508bfb61c799]) successfully deleted
I20240503 02:00:06.811844 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:06.818107 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:06.820328 6861 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:00:06.832684 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:06.840684 9474 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:00:06.843768 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:06.866504 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:06.884708 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:06.971906 7012 consensus_queue.cc:237] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:06.973479 6882 raft_consensus.cc:1261] T 5e20b869a89e44f1b0ba854163c31f06 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:06.973515 7145 raft_consensus.cc:1261] T 5e20b869a89e44f1b0ba854163c31f06 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:06.973917 18534 consensus_queue.cc:1035] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:06.974124 18779 consensus_queue.cc:1035] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:06.977180 18534 raft_consensus.cc:2932] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:06.977967 6882 raft_consensus.cc:2932] T 5e20b869a89e44f1b0ba854163c31f06 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:06.980427 7143 raft_consensus.cc:2932] T 5e20b869a89e44f1b0ba854163c31f06 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:06.981618 6736 catalog_manager.cc:5484] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:06.985616 9341 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e20b869a89e44f1b0ba854163c31f06 with delete_type TABLET_DATA_TOMBSTONED (TS 12e8c683db054df3b75c6285b98a5ad8 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:36040
I20240503 02:00:06.985960 18713 tablet_replica.cc:330] T 5e20b869a89e44f1b0ba854163c31f06 P 12e8c683db054df3b75c6285b98a5ad8: stopping tablet replica
I20240503 02:00:06.986150 18713 raft_consensus.cc:2229] T 5e20b869a89e44f1b0ba854163c31f06 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:06.986295 18713 raft_consensus.cc:2258] T 5e20b869a89e44f1b0ba854163c31f06 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:06.986775 18713 ts_tablet_manager.cc:1875] T 5e20b869a89e44f1b0ba854163c31f06 P 12e8c683db054df3b75c6285b98a5ad8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:06.989674 18713 ts_tablet_manager.cc:1888] T 5e20b869a89e44f1b0ba854163c31f06 P 12e8c683db054df3b75c6285b98a5ad8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:00:06.989769 18713 log.cc:1196] T 5e20b869a89e44f1b0ba854163c31f06 P 12e8c683db054df3b75c6285b98a5ad8: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/5e20b869a89e44f1b0ba854163c31f06
I20240503 02:00:06.990796 6727 catalog_manager.cc:4830] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): tablet 5e20b869a89e44f1b0ba854163c31f06 (table rebalancer_extra_table_50 [id=6f17b2a98eb44079957e3e2ae659fb2c]) successfully deleted
I20240503 02:00:07.214334 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:07.216915 6861 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:00:07.221277 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:07.223986 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:07.226750 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:07.259048 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:07.280671 9474 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:00:07.299124 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:07.413877 9626 consensus_queue.cc:237] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:07.415657 7012 raft_consensus.cc:1261] T b93fe374d2a9496a847480b7824a5937 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer b7062fbbc2b54fff9e99ec4ea5562c76: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:07.415657 9757 raft_consensus.cc:1261] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer b7062fbbc2b54fff9e99ec4ea5562c76: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:07.416121 16102 consensus_queue.cc:1035] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:07.416349 18142 consensus_queue.cc:1035] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:07.418659 18853 raft_consensus.cc:2932] T b93fe374d2a9496a847480b7824a5937 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:07.419114 7012 raft_consensus.cc:2932] T b93fe374d2a9496a847480b7824a5937 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:07.419263 9757 raft_consensus.cc:2932] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:07.426048 6739 catalog_manager.cc:5484] T b93fe374d2a9496a847480b7824a5937 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:07.430361 9474 tablet_service.cc:1514] Processing DeleteTablet for tablet b93fe374d2a9496a847480b7824a5937 with delete_type TABLET_DATA_TOMBSTONED (TS bb5364d730cb4489a0bfc6c270b7450a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54678
I20240503 02:00:07.432647 18858 tablet_replica.cc:330] T b93fe374d2a9496a847480b7824a5937 P bb5364d730cb4489a0bfc6c270b7450a: stopping tablet replica
I20240503 02:00:07.432879 18858 raft_consensus.cc:2229] T b93fe374d2a9496a847480b7824a5937 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:07.433068 18858 raft_consensus.cc:2258] T b93fe374d2a9496a847480b7824a5937 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:07.433630 18858 ts_tablet_manager.cc:1875] T b93fe374d2a9496a847480b7824a5937 P bb5364d730cb4489a0bfc6c270b7450a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:07.442875 18858 ts_tablet_manager.cc:1888] T b93fe374d2a9496a847480b7824a5937 P bb5364d730cb4489a0bfc6c270b7450a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:00:07.443037 18858 log.cc:1196] T b93fe374d2a9496a847480b7824a5937 P bb5364d730cb4489a0bfc6c270b7450a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/b93fe374d2a9496a847480b7824a5937
I20240503 02:00:07.444636 6725 catalog_manager.cc:4830] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): tablet b93fe374d2a9496a847480b7824a5937 (table rebalancer_extra_table_26 [id=83830de54936404aa798ca5e4773319d]) successfully deleted
I20240503 02:00:07.680445 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:07.686434 6861 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:00:07.688210 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:07.688701 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:07.699764 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:07.699810 9474 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:00:07.709947 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:07.731938 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:07.834986 9493 consensus_queue.cc:237] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:07.837136 9363 raft_consensus.cc:1261] T 5089809a9f3846f5b30397b1062d4e55 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:07.837723 6883 raft_consensus.cc:1261] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:07.838675 18921 consensus_queue.cc:1035] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:07.839165 16828 consensus_queue.cc:1035] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:07.842589 18921 raft_consensus.cc:2932] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:07.843039 9363 raft_consensus.cc:2932] T 5089809a9f3846f5b30397b1062d4e55 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:07.843658 6883 raft_consensus.cc:2932] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:07.847779 6740 catalog_manager.cc:5484] T 5089809a9f3846f5b30397b1062d4e55 P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:07.851610 9605 tablet_service.cc:1514] Processing DeleteTablet for tablet 5089809a9f3846f5b30397b1062d4e55 with delete_type TABLET_DATA_TOMBSTONED (TS b7062fbbc2b54fff9e99ec4ea5562c76 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:58562
I20240503 02:00:07.854794 18941 tablet_replica.cc:330] T 5089809a9f3846f5b30397b1062d4e55 P b7062fbbc2b54fff9e99ec4ea5562c76: stopping tablet replica
I20240503 02:00:07.855090 18941 raft_consensus.cc:2229] T 5089809a9f3846f5b30397b1062d4e55 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:07.855460 18941 raft_consensus.cc:2258] T 5089809a9f3846f5b30397b1062d4e55 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:07.856079 18941 ts_tablet_manager.cc:1875] T 5089809a9f3846f5b30397b1062d4e55 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:07.859913 18941 ts_tablet_manager.cc:1888] T 5089809a9f3846f5b30397b1062d4e55 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:00:07.860071 18941 log.cc:1196] T 5089809a9f3846f5b30397b1062d4e55 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/5089809a9f3846f5b30397b1062d4e55
I20240503 02:00:07.860795 6727 catalog_manager.cc:4830] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): tablet 5089809a9f3846f5b30397b1062d4e55 (table rebalancer_extra_table_40 [id=28d2e532bc114d04b0b1e87094387600]) successfully deleted
I20240503 02:00:08.074795 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:08.076763 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:08.100140 6861 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:00:08.105531 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:08.105803 9474 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:00:08.123553 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:08.156989 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:08.184659 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:08.332993 7275 consensus_queue.cc:237] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:08.336256 9493 raft_consensus.cc:1261] T e811662836164c3fb29a1b12ace7eb41 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:08.338306 9363 raft_consensus.cc:1261] T e811662836164c3fb29a1b12ace7eb41 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:08.339587 17364 consensus_queue.cc:1035] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:08.348394 18721 consensus_queue.cc:1035] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:08.351182 17364 raft_consensus.cc:2932] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:08.355921 6741 catalog_manager.cc:5484] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: config changed from index 6 to 7, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:08.356729 9493 raft_consensus.cc:2932] T e811662836164c3fb29a1b12ace7eb41 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:08.358395 9362 raft_consensus.cc:2932] T e811662836164c3fb29a1b12ace7eb41 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:08.384198 7123 tablet_service.cc:1514] Processing DeleteTablet for tablet e811662836164c3fb29a1b12ace7eb41 with delete_type TABLET_DATA_TOMBSTONED (TS c688f3df56ac4b6d9b0c30fa4cc8ca2a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50370
I20240503 02:00:08.389451 19025 tablet_replica.cc:330] T e811662836164c3fb29a1b12ace7eb41 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 02:00:08.389741 19025 raft_consensus.cc:2229] T e811662836164c3fb29a1b12ace7eb41 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:08.390012 19025 raft_consensus.cc:2258] T e811662836164c3fb29a1b12ace7eb41 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:08.390686 19025 ts_tablet_manager.cc:1875] T e811662836164c3fb29a1b12ace7eb41 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:08.395120 19025 ts_tablet_manager.cc:1888] T e811662836164c3fb29a1b12ace7eb41 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:00:08.395772 19025 log.cc:1196] T e811662836164c3fb29a1b12ace7eb41 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/e811662836164c3fb29a1b12ace7eb41
I20240503 02:00:08.399607 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet e811662836164c3fb29a1b12ace7eb41 (table rebalance_test_table_17 [id=b061a61957e74150a4af26c16c277330]) successfully deleted
I20240503 02:00:08.645007 6861 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:00:08.645128 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:08.660439 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:08.673782 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:08.679157 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:08.686002 9474 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:00:08.699518 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:08.704701 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:08.828413 6883 consensus_queue.cc:237] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:08.829943 9624 raft_consensus.cc:1261] T 4bd8226227c348d5a5d4d8ff28959d27 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:08.829943 9757 raft_consensus.cc:1261] T 4bd8226227c348d5a5d4d8ff28959d27 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:08.830482 19014 consensus_queue.cc:1035] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:08.830904 18517 consensus_queue.cc:1035] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:08.833117 18517 raft_consensus.cc:2932] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:08.833756 9757 raft_consensus.cc:2932] T 4bd8226227c348d5a5d4d8ff28959d27 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:08.837460 9624 raft_consensus.cc:2932] T 4bd8226227c348d5a5d4d8ff28959d27 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:08.840881 6739 catalog_manager.cc:5484] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index 6 to 7, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:08.846084 7254 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bd8226227c348d5a5d4d8ff28959d27 with delete_type TABLET_DATA_TOMBSTONED (TS 1d0e738c34af40fa9950c6f086619b3a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45608
I20240503 02:00:08.848863 19119 tablet_replica.cc:330] T 4bd8226227c348d5a5d4d8ff28959d27 P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 02:00:08.850224 19119 raft_consensus.cc:2229] T 4bd8226227c348d5a5d4d8ff28959d27 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:08.850466 19119 raft_consensus.cc:2258] T 4bd8226227c348d5a5d4d8ff28959d27 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:08.851280 19119 ts_tablet_manager.cc:1875] T 4bd8226227c348d5a5d4d8ff28959d27 P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:08.855005 19119 ts_tablet_manager.cc:1888] T 4bd8226227c348d5a5d4d8ff28959d27 P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:00:08.855140 19119 log.cc:1196] T 4bd8226227c348d5a5d4d8ff28959d27 P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/4bd8226227c348d5a5d4d8ff28959d27
I20240503 02:00:08.857268 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet 4bd8226227c348d5a5d4d8ff28959d27 (table rebalance_test_table_3 [id=45e94c90bda847688db4572a4f9d0c02]) successfully deleted
I20240503 02:00:09.030871 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:09.032420 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:09.033764 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:09.048841 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:09.058745 9474 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:00:09.066479 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:09.067219 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:09.095633 6861 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:00:09.203140 9755 consensus_queue.cc:237] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:09.205483 7275 raft_consensus.cc:1261] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.206020 18995 consensus_queue.cc:1035] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:09.206151 7145 raft_consensus.cc:1261] T 0bbbd17c78bc4e329a3a7c7008a34d86 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.206687 18995 consensus_queue.cc:1035] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:09.208097 18996 raft_consensus.cc:2932] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:09.208549 7275 raft_consensus.cc:2932] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:09.211800 7145 raft_consensus.cc:2932] T 0bbbd17c78bc4e329a3a7c7008a34d86 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:09.212420 6736 catalog_manager.cc:5484] T 0bbbd17c78bc4e329a3a7c7008a34d86 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:09.215754 9341 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bbbd17c78bc4e329a3a7c7008a34d86 with delete_type TABLET_DATA_TOMBSTONED (TS 12e8c683db054df3b75c6285b98a5ad8 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:36040
I20240503 02:00:09.218739 19186 tablet_replica.cc:330] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 12e8c683db054df3b75c6285b98a5ad8: stopping tablet replica
I20240503 02:00:09.219096 19186 raft_consensus.cc:2229] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:09.219309 19186 raft_consensus.cc:2258] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:09.220247 19186 ts_tablet_manager.cc:1875] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 12e8c683db054df3b75c6285b98a5ad8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:09.224768 19186 ts_tablet_manager.cc:1888] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 12e8c683db054df3b75c6285b98a5ad8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:00:09.224990 19186 log.cc:1196] T 0bbbd17c78bc4e329a3a7c7008a34d86 P 12e8c683db054df3b75c6285b98a5ad8: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/0bbbd17c78bc4e329a3a7c7008a34d86
I20240503 02:00:09.225641 6727 catalog_manager.cc:4830] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): tablet 0bbbd17c78bc4e329a3a7c7008a34d86 (table rebalancer_extra_table_36 [id=9c1714bb582349bcb6ea24e634fcd46c]) successfully deleted
I20240503 02:00:09.419996 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:09.422717 6861 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:00:09.426455 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:09.432512 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:09.439517 9474 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:00:09.442314 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:09.445055 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:09.466794 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:09.555593 9362 consensus_queue.cc:237] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:09.557047 7012 raft_consensus.cc:1261] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.557272 9755 raft_consensus.cc:1261] T 6fd0378b2cff42a681361930f487d834 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.557518 18823 consensus_queue.cc:1035] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:09.557883 18689 consensus_queue.cc:1035] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:09.559603 18689 raft_consensus.cc:2932] T 6fd0378b2cff42a681361930f487d834 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:09.559952 7012 raft_consensus.cc:2932] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:09.564884 6732 catalog_manager.cc:5484] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:09.567337 9757 raft_consensus.cc:2932] T 6fd0378b2cff42a681361930f487d834 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:09.569123 9474 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fd0378b2cff42a681361930f487d834 with delete_type TABLET_DATA_TOMBSTONED (TS bb5364d730cb4489a0bfc6c270b7450a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54678
I20240503 02:00:09.575328 19275 tablet_replica.cc:330] T 6fd0378b2cff42a681361930f487d834 P bb5364d730cb4489a0bfc6c270b7450a: stopping tablet replica
I20240503 02:00:09.575608 19275 raft_consensus.cc:2229] T 6fd0378b2cff42a681361930f487d834 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:09.575831 19275 raft_consensus.cc:2258] T 6fd0378b2cff42a681361930f487d834 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:09.576284 19275 ts_tablet_manager.cc:1875] T 6fd0378b2cff42a681361930f487d834 P bb5364d730cb4489a0bfc6c270b7450a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:09.582075 19275 ts_tablet_manager.cc:1888] T 6fd0378b2cff42a681361930f487d834 P bb5364d730cb4489a0bfc6c270b7450a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:00:09.582190 19275 log.cc:1196] T 6fd0378b2cff42a681361930f487d834 P bb5364d730cb4489a0bfc6c270b7450a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/6fd0378b2cff42a681361930f487d834
I20240503 02:00:09.583289 6725 catalog_manager.cc:4830] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): tablet 6fd0378b2cff42a681361930f487d834 (table rebalancer_extra_table_34 [id=cb5e07c45e8b46888678a15d79f2cfd9]) successfully deleted
I20240503 02:00:09.741645 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:09.753840 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:09.756484 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:09.756783 9474 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:00:09.771430 6861 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:00:09.774222 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:09.779541 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:09.792932 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:09.881363 9362 consensus_queue.cc:237] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:09.883085 9493 raft_consensus.cc:1261] T 9b8976486acf49d5a9d0d496379ee2a6 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.883293 6882 raft_consensus.cc:1261] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:09.883937 18689 consensus_queue.cc:1035] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:09.885082 18823 consensus_queue.cc:1035] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:09.887554 18823 raft_consensus.cc:2932] T 9b8976486acf49d5a9d0d496379ee2a6 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:09.888314 6882 raft_consensus.cc:2932] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:09.889616 9494 raft_consensus.cc:2932] T 9b8976486acf49d5a9d0d496379ee2a6 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:09.893627 6741 catalog_manager.cc:5484] T 9b8976486acf49d5a9d0d496379ee2a6 P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:09.897486 9605 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b8976486acf49d5a9d0d496379ee2a6 with delete_type TABLET_DATA_TOMBSTONED (TS b7062fbbc2b54fff9e99ec4ea5562c76 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:58562
I20240503 02:00:09.897991 19352 tablet_replica.cc:330] T 9b8976486acf49d5a9d0d496379ee2a6 P b7062fbbc2b54fff9e99ec4ea5562c76: stopping tablet replica
I20240503 02:00:09.898200 19352 raft_consensus.cc:2229] T 9b8976486acf49d5a9d0d496379ee2a6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:09.898370 19352 raft_consensus.cc:2258] T 9b8976486acf49d5a9d0d496379ee2a6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:09.898908 19352 ts_tablet_manager.cc:1875] T 9b8976486acf49d5a9d0d496379ee2a6 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:09.902443 19352 ts_tablet_manager.cc:1888] T 9b8976486acf49d5a9d0d496379ee2a6 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:00:09.904649 19352 log.cc:1196] T 9b8976486acf49d5a9d0d496379ee2a6 P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/9b8976486acf49d5a9d0d496379ee2a6
I20240503 02:00:09.906208 6727 catalog_manager.cc:4830] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): tablet 9b8976486acf49d5a9d0d496379ee2a6 (table rebalancer_extra_table_44 [id=b15f374995474d32bb2e755f0bcb94a5]) successfully deleted
I20240503 02:00:10.058777 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:10.066174 9474 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:00:10.066574 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:10.078963 6861 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:00:10.083475 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:10.091658 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:10.107597 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:10.115638 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:10.215461 7012 consensus_queue.cc:237] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:10.217057 7274 raft_consensus.cc:1261] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:10.217353 9363 raft_consensus.cc:1261] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:10.217530 18937 consensus_queue.cc:1035] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:10.217949 18937 consensus_queue.cc:1035] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:10.219992 18534 raft_consensus.cc:2932] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:10.222692 9363 raft_consensus.cc:2932] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:10.223266 7274 raft_consensus.cc:2932] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:10.225508 6735 catalog_manager.cc:5484] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index 2 to 3, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:10.228890 9735 tablet_service.cc:1514] Processing DeleteTablet for tablet aaeebd4c884445ba9f44ac9bfcb9c7ba with delete_type TABLET_DATA_TOMBSTONED (TS db1a0000aaa346f79c0aa4b6c5105191 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:48152
I20240503 02:00:10.230940 19420 tablet_replica.cc:330] T aaeebd4c884445ba9f44ac9bfcb9c7ba P db1a0000aaa346f79c0aa4b6c5105191: stopping tablet replica
I20240503 02:00:10.232421 19420 raft_consensus.cc:2229] T aaeebd4c884445ba9f44ac9bfcb9c7ba P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:10.232688 19420 raft_consensus.cc:2258] T aaeebd4c884445ba9f44ac9bfcb9c7ba P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:10.233285 19420 ts_tablet_manager.cc:1875] T aaeebd4c884445ba9f44ac9bfcb9c7ba P db1a0000aaa346f79c0aa4b6c5105191: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:10.236716 19420 ts_tablet_manager.cc:1888] T aaeebd4c884445ba9f44ac9bfcb9c7ba P db1a0000aaa346f79c0aa4b6c5105191: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:00:10.236872 19420 log.cc:1196] T aaeebd4c884445ba9f44ac9bfcb9c7ba P db1a0000aaa346f79c0aa4b6c5105191: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/aaeebd4c884445ba9f44ac9bfcb9c7ba
I20240503 02:00:10.238171 6726 catalog_manager.cc:4830] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): tablet aaeebd4c884445ba9f44ac9bfcb9c7ba (table rebalancer_extra_table_18 [id=9c9f7f12e036416d8388508bfb61c799]) successfully deleted
I20240503 02:00:10.414052 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:10.422564 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:10.422587 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:10.425238 6861 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:00:10.433240 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:10.446405 9474 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:00:10.455531 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:10.456444 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:10.541388 7145 consensus_queue.cc:237] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:10.543026 9757 raft_consensus.cc:1261] T 8ca4e33409b74f359d3242248652d8bb P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:10.543027 9624 raft_consensus.cc:1261] T 8ca4e33409b74f359d3242248652d8bb P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:10.543463 16780 consensus_queue.cc:1035] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:10.543738 19281 consensus_queue.cc:1035] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:10.545647 19281 raft_consensus.cc:2932] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:10.546002 9624 raft_consensus.cc:2932] T 8ca4e33409b74f359d3242248652d8bb P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:10.551497 6734 catalog_manager.cc:5484] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index 2 to 3, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:10.554270 9757 raft_consensus.cc:2932] T 8ca4e33409b74f359d3242248652d8bb P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:10.556284 7254 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ca4e33409b74f359d3242248652d8bb with delete_type TABLET_DATA_TOMBSTONED (TS 1d0e738c34af40fa9950c6f086619b3a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45608
I20240503 02:00:10.558961 19513 tablet_replica.cc:330] T 8ca4e33409b74f359d3242248652d8bb P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 02:00:10.559162 19513 raft_consensus.cc:2229] T 8ca4e33409b74f359d3242248652d8bb P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:10.559576 19513 raft_consensus.cc:2258] T 8ca4e33409b74f359d3242248652d8bb P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:10.560276 19513 ts_tablet_manager.cc:1875] T 8ca4e33409b74f359d3242248652d8bb P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:10.563213 19513 ts_tablet_manager.cc:1888] T 8ca4e33409b74f359d3242248652d8bb P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:00:10.563330 19513 log.cc:1196] T 8ca4e33409b74f359d3242248652d8bb P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/8ca4e33409b74f359d3242248652d8bb
I20240503 02:00:10.564000 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet 8ca4e33409b74f359d3242248652d8bb (table rebalancer_extra_table_46 [id=bc4eab53df254ef4bc9747cb965654a7]) successfully deleted
I20240503 02:00:10.956514 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:10.957021 6861 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:00:10.964685 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:10.969133 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:10.969336 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:10.977883 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:10.979226 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:11.008652 9474 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:00:11.053493 9494 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3e824fcc9aac449d9373655b304010e2"
dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41986
I20240503 02:00:11.053799 9494 raft_consensus.cc:595] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Received request to transfer leadership
I20240503 02:00:11.212291 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:11.225195 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:11.225287 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:11.229795 9474 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:00:11.236608 6861 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:00:11.251861 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:11.256042 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:11.260128 19493 raft_consensus.cc:979] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a: : Instructing follower 5dcc33cc621a4cbbae155f3230fb7c46 to start an election
I20240503 02:00:11.260237 19493 raft_consensus.cc:1067] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Signalling peer 5dcc33cc621a4cbbae155f3230fb7c46 to start an election
I20240503 02:00:11.261135 6883 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3e824fcc9aac449d9373655b304010e2"
dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
from {username='slave'} at 127.0.99.134:59615
I20240503 02:00:11.261289 6883 raft_consensus.cc:482] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:00:11.261354 6883 raft_consensus.cc:3037] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:11.263267 6883 raft_consensus.cc:504] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:11.264545 6883 leader_election.cc:288] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 election: Requested vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 02:00:11.264605 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e824fcc9aac449d9373655b304010e2" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 02:00:11.264747 7014 raft_consensus.cc:3037] T 3e824fcc9aac449d9373655b304010e2 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:11.265817 7014 raft_consensus.cc:2454] T 3e824fcc9aac449d9373655b304010e2 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 2.
I20240503 02:00:11.266451 9493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e824fcc9aac449d9373655b304010e2" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "bb5364d730cb4489a0bfc6c270b7450a"
I20240503 02:00:11.266620 9493 raft_consensus.cc:3032] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:00:11.266666 9493 raft_consensus.cc:729] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Running, Role: LEADER
I20240503 02:00:11.266839 9493 consensus_queue.cc:260] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:11.267036 9493 raft_consensus.cc:3037] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:11.267673 9362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e824fcc9aac449d9373655b304010e2" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "12e8c683db054df3b75c6285b98a5ad8"
I20240503 02:00:11.267802 9362 raft_consensus.cc:3037] T 3e824fcc9aac449d9373655b304010e2 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:11.268146 9493 raft_consensus.cc:2454] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 2.
I20240503 02:00:11.269289 9362 raft_consensus.cc:2454] T 3e824fcc9aac449d9373655b304010e2 P 12e8c683db054df3b75c6285b98a5ad8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 2.
I20240503 02:00:11.269989 6818 leader_election.cc:302] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 12e8c683db054df3b75c6285b98a5ad8, 5dcc33cc621a4cbbae155f3230fb7c46, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 02:00:11.270291 19481 raft_consensus.cc:2781] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:11.270411 19481 raft_consensus.cc:686] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 2 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 02:00:11.270788 19481 consensus_queue.cc:237] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:11.273919 6732 catalog_manager.cc:5484] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 1 to 2, leader changed from bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:11.284751 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:11.495364 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:11.495565 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:11.496624 6861 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:11.514284 9474 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:11.514645 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:11.516520 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:11.536581 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:11.570075 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:11.621770 9363 consensus_queue.cc:237] T fd29f3b059874b019153b64ef8aaaa48 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:11.623351 7014 raft_consensus.cc:1261] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:00:11.623366 9626 raft_consensus.cc:1261] T fd29f3b059874b019153b64ef8aaaa48 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 2 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:00:11.623744 18689 consensus_queue.cc:1035] T fd29f3b059874b019153b64ef8aaaa48 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:11.623909 18823 consensus_queue.cc:1035] T fd29f3b059874b019153b64ef8aaaa48 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:11.626689 18689 raft_consensus.cc:2932] T fd29f3b059874b019153b64ef8aaaa48 P 12e8c683db054df3b75c6285b98a5ad8 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:11.627223 7014 raft_consensus.cc:2932] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:11.632251 6732 catalog_manager.cc:5484] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index 2 to 4, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New cstate: current_term: 2 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:11.632673 9626 raft_consensus.cc:2932] T fd29f3b059874b019153b64ef8aaaa48 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:11.635489 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet fd29f3b059874b019153b64ef8aaaa48 with delete_type TABLET_DATA_TOMBSTONED (TS 5dcc33cc621a4cbbae155f3230fb7c46 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35310
I20240503 02:00:11.635883 19703 tablet_replica.cc:330] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 02:00:11.636117 19703 raft_consensus.cc:2229] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:11.636318 19703 raft_consensus.cc:2258] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:11.636880 19703 ts_tablet_manager.cc:1875] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:11.639470 19703 ts_tablet_manager.cc:1888] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 02:00:11.639546 19703 log.cc:1196] T fd29f3b059874b019153b64ef8aaaa48 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/fd29f3b059874b019153b64ef8aaaa48
I20240503 02:00:11.640211 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet fd29f3b059874b019153b64ef8aaaa48 (table rebalancer_extra_table_52 [id=7084fdc09131424f92874f08cc5ef36b]) successfully deleted
I20240503 02:00:11.717114 9493 raft_consensus.cc:1261] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [term 2 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:00:11.717671 19481 consensus_queue.cc:1035] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:11.808466 7014 raft_consensus.cc:1261] T 3e824fcc9aac449d9373655b304010e2 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:00:11.809305 19481 consensus_queue.cc:1035] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:11.810495 9363 consensus_queue.cc:237] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:11.812489 7145 raft_consensus.cc:1261] T c4930b3ca84749f6a4cd245411f66075 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:11.812642 9298 consensus_peers.cc:485] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c4930b3ca84749f6a4cd245411f66075. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:11.812932 18689 consensus_queue.cc:1035] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:11.813071 9757 raft_consensus.cc:1261] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:11.813457 18689 consensus_queue.cc:1035] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:11.815740 9363 raft_consensus.cc:1261] T 3e824fcc9aac449d9373655b304010e2 P 12e8c683db054df3b75c6285b98a5ad8 [term 2 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:00:11.816166 19480 consensus_queue.cc:1035] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:00:11.816264 18689 raft_consensus.cc:2932] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:11.818928 7145 raft_consensus.cc:2932] T c4930b3ca84749f6a4cd245411f66075 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:11.819661 9757 raft_consensus.cc:2932] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:11.826717 6735 catalog_manager.cc:5484] T c4930b3ca84749f6a4cd245411f66075 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index -1 to 2, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:11.852170 7145 consensus_queue.cc:237] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:11.853844 9493 raft_consensus.cc:1261] T 75175ed25db846b9ba8e8e7b595102ca P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:11.854267 9757 raft_consensus.cc:1261] T 75175ed25db846b9ba8e8e7b595102ca P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:11.854295 19512 consensus_queue.cc:1035] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:00:11.854713 7079 consensus_peers.cc:485] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): Couldn't send request to peer 1d0e738c34af40fa9950c6f086619b3a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75175ed25db846b9ba8e8e7b595102ca. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:11.854929 19512 consensus_queue.cc:1035] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:11.857105 19281 raft_consensus.cc:2932] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:11.857775 9493 raft_consensus.cc:2932] T 75175ed25db846b9ba8e8e7b595102ca P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:11.859510 9757 raft_consensus.cc:2932] T 75175ed25db846b9ba8e8e7b595102ca P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:11.863596 6735 catalog_manager.cc:5484] T 75175ed25db846b9ba8e8e7b595102ca P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:11.888947 9757 consensus_queue.cc:237] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:11.890627 9626 raft_consensus.cc:1261] T 8feff37a8a6043d2b2f7581deb7c7934 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:11.890653 9363 raft_consensus.cc:1261] T 8feff37a8a6043d2b2f7581deb7c7934 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:11.891027 19626 consensus_queue.cc:1035] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:11.891201 19514 consensus_queue.cc:1035] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:00:11.891352 9690 consensus_peers.cc:485] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): Couldn't send request to peer 63331f455be049c1bdd6c23c59ac1bef. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8feff37a8a6043d2b2f7581deb7c7934. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:11.893445 19514 raft_consensus.cc:2932] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:11.894047 9363 raft_consensus.cc:2932] T 8feff37a8a6043d2b2f7581deb7c7934 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:11.898059 6732 catalog_manager.cc:5484] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:11.900626 9626 raft_consensus.cc:2932] T 8feff37a8a6043d2b2f7581deb7c7934 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:11.931845 9363 consensus_queue.cc:237] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:11.933882 9626 raft_consensus.cc:1261] T beb175f81d9e48229466503ebf7e3c09 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:11.933883 9494 raft_consensus.cc:1261] T beb175f81d9e48229466503ebf7e3c09 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:11.934391 9299 consensus_peers.cc:485] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8 -> Peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): Couldn't send request to peer 5dcc33cc621a4cbbae155f3230fb7c46. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: beb175f81d9e48229466503ebf7e3c09. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:11.934401 18689 consensus_queue.cc:1035] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:11.934840 19817 consensus_queue.cc:1035] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:11.936647 19817 raft_consensus.cc:2932] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:11.937039 9494 raft_consensus.cc:2932] T beb175f81d9e48229466503ebf7e3c09 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:11.939376 9626 raft_consensus.cc:2932] T beb175f81d9e48229466503ebf7e3c09 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:11.942106 6732 catalog_manager.cc:5484] T beb175f81d9e48229466503ebf7e3c09 P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:11.965786 7274 consensus_queue.cc:237] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:11.967446 9757 raft_consensus.cc:1261] T d2fbadb911aa4bd7b4db021860c202d3 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:11.967677 7145 raft_consensus.cc:1261] T d2fbadb911aa4bd7b4db021860c202d3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:11.967881 19894 consensus_queue.cc:1035] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 02:00:11.968375 7211 consensus_peers.cc:485] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d2fbadb911aa4bd7b4db021860c202d3. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:11.968544 19894 consensus_queue.cc:1035] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:11.970837 19556 raft_consensus.cc:2932] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:11.971217 7143 raft_consensus.cc:2932] T d2fbadb911aa4bd7b4db021860c202d3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:11.974324 9757 raft_consensus.cc:2932] T d2fbadb911aa4bd7b4db021860c202d3 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:11.975803 6735 catalog_manager.cc:5484] T d2fbadb911aa4bd7b4db021860c202d3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index 5 to 6, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:12.001653 7143 consensus_queue.cc:237] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:12.003019 7014 raft_consensus.cc:1261] T 7672f42acd9446ef85eced07729c197b P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.003151 9757 raft_consensus.cc:1261] T 7672f42acd9446ef85eced07729c197b P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:12.003299 7080 consensus_peers.cc:485] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7672f42acd9446ef85eced07729c197b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:12.003459 19839 consensus_queue.cc:1035] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:12.003630 19683 consensus_queue.cc:1035] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:12.005890 19683 raft_consensus.cc:2932] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:12.006348 7014 raft_consensus.cc:2932] T 7672f42acd9446ef85eced07729c197b P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:12.008303 9757 raft_consensus.cc:2932] T 7672f42acd9446ef85eced07729c197b P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:12.011246 6736 catalog_manager.cc:5484] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index -1 to 2, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:12.041816 6883 consensus_queue.cc:237] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:12.043669 7274 raft_consensus.cc:1261] T 2084fee295274ad89b407a933fb6070c P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.043669 9755 raft_consensus.cc:1261] T 2084fee295274ad89b407a933fb6070c P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:12.043677 6816 consensus_peers.cc:485] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2084fee295274ad89b407a933fb6070c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:12.044090 19480 consensus_queue.cc:1035] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:12.044468 19480 consensus_queue.cc:1035] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:12.047680 19480 raft_consensus.cc:2932] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 02:00:12.048300 9755 raft_consensus.cc:2932] T 2084fee295274ad89b407a933fb6070c P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 02:00:12.051983 7274 raft_consensus.cc:2932] T 2084fee295274ad89b407a933fb6070c P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 02:00:12.052114 6735 catalog_manager.cc:5484] T 2084fee295274ad89b407a933fb6070c P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: config changed from index -1 to 2, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 02:00:12.076980 9362 consensus_queue.cc:237] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
W20240503 02:00:12.078728 9298 consensus_peers.cc:485] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8 -> Peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): Couldn't send request to peer c688f3df56ac4b6d9b0c30fa4cc8ca2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2755530dc6f04baaaafd5d5214b150af. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:12.078755 9626 raft_consensus.cc:1261] T 2755530dc6f04baaaafd5d5214b150af P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.078881 7014 raft_consensus.cc:1261] T 2755530dc6f04baaaafd5d5214b150af P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.079174 18823 consensus_queue.cc:1035] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:12.079407 19817 consensus_queue.cc:1035] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:12.081444 18689 raft_consensus.cc:2932] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:12.081964 9626 raft_consensus.cc:2932] T 2755530dc6f04baaaafd5d5214b150af P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:12.084416 7014 raft_consensus.cc:2932] T 2755530dc6f04baaaafd5d5214b150af P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:12.085718 6735 catalog_manager.cc:5484] T 2755530dc6f04baaaafd5d5214b150af P b7062fbbc2b54fff9e99ec4ea5562c76 reported cstate change: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:12.145272 7143 consensus_queue.cc:237] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:12.147066 9626 raft_consensus.cc:1261] T 8295ea038a88485095544cc608b000c2 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 02:00:12.147377 7079 consensus_peers.cc:485] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): Couldn't send request to peer db1a0000aaa346f79c0aa4b6c5105191. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8295ea038a88485095544cc608b000c2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:12.147596 19839 consensus_queue.cc:1035] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:12.148202 6883 raft_consensus.cc:1261] T 8295ea038a88485095544cc608b000c2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:12.148643 19839 consensus_queue.cc:1035] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:12.150772 19281 raft_consensus.cc:2932] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:12.151605 6883 raft_consensus.cc:2932] T 8295ea038a88485095544cc608b000c2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:12.155015 9626 raft_consensus.cc:2932] T 8295ea038a88485095544cc608b000c2 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:12.155886 6732 catalog_manager.cc:5484] T 8295ea038a88485095544cc608b000c2 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index 5 to 6, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:12.178614 6882 consensus_queue.cc:237] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:12.180116 9626 raft_consensus.cc:1261] T 6364f3919c42498ca47ba114d525279b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:12.180260 6817 consensus_peers.cc:485] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): Couldn't send request to peer 1d0e738c34af40fa9950c6f086619b3a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6364f3919c42498ca47ba114d525279b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:12.180285 9362 raft_consensus.cc:1261] T 6364f3919c42498ca47ba114d525279b P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.180581 19934 consensus_queue.cc:1035] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:12.180766 19480 consensus_queue.cc:1035] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:12.182917 19480 raft_consensus.cc:2932] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:12.185725 9626 raft_consensus.cc:2932] T 6364f3919c42498ca47ba114d525279b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:12.186915 9362 raft_consensus.cc:2932] T 6364f3919c42498ca47ba114d525279b P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:12.187485 6732 catalog_manager.cc:5484] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:12.212684 7274 consensus_queue.cc:237] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:12.214022 6882 raft_consensus.cc:1261] T aab7d8c559ff446192f471aacaa47584 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.214426 19894 consensus_queue.cc:1035] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:12.214510 9362 raft_consensus.cc:1261] T aab7d8c559ff446192f471aacaa47584 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:12.214790 7209 consensus_peers.cc:485] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): Couldn't send request to peer 63331f455be049c1bdd6c23c59ac1bef. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aab7d8c559ff446192f471aacaa47584. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:12.214915 19894 consensus_queue.cc:1035] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:12.217872 17364 raft_consensus.cc:2932] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:12.218382 9362 raft_consensus.cc:2932] T aab7d8c559ff446192f471aacaa47584 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:12.221417 6882 raft_consensus.cc:2932] T aab7d8c559ff446192f471aacaa47584 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:12.221930 6736 catalog_manager.cc:5484] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:12.252956 9626 consensus_queue.cc:237] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:12.254900 9494 raft_consensus.cc:1261] T d4fa0337228d4f8cb6b6c3aab437619e P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer b7062fbbc2b54fff9e99ec4ea5562c76: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:12.254949 9561 consensus_peers.cc:485] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76 -> Peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): Couldn't send request to peer c688f3df56ac4b6d9b0c30fa4cc8ca2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4fa0337228d4f8cb6b6c3aab437619e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:12.255115 7012 raft_consensus.cc:1261] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer b7062fbbc2b54fff9e99ec4ea5562c76: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.255410 20025 consensus_queue.cc:1035] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:12.255597 18142 consensus_queue.cc:1035] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:12.258462 20025 raft_consensus.cc:2932] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:12.259433 9494 raft_consensus.cc:2932] T d4fa0337228d4f8cb6b6c3aab437619e P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:12.264052 6736 catalog_manager.cc:5484] T d4fa0337228d4f8cb6b6c3aab437619e P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:12.264132 7012 raft_consensus.cc:2932] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:12.295094 9626 consensus_queue.cc:237] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:12.297250 9494 raft_consensus.cc:1261] T 944657ceab07465092a950a8c793ceff P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer b7062fbbc2b54fff9e99ec4ea5562c76: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:12.297359 9560 consensus_peers.cc:485] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76 -> Peer db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): Couldn't send request to peer db1a0000aaa346f79c0aa4b6c5105191. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 944657ceab07465092a950a8c793ceff. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:12.297632 19953 consensus_queue.cc:1035] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:12.297722 6882 raft_consensus.cc:1261] T 944657ceab07465092a950a8c793ceff P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer b7062fbbc2b54fff9e99ec4ea5562c76: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.298056 19953 consensus_queue.cc:1035] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:12.299868 20025 raft_consensus.cc:2932] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:12.300328 6882 raft_consensus.cc:2932] T 944657ceab07465092a950a8c793ceff P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:12.307581 6735 catalog_manager.cc:5484] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76 reported cstate change: config changed from index -1 to 2, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:12.308810 9494 raft_consensus.cc:2932] T 944657ceab07465092a950a8c793ceff P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:12.342428 7014 consensus_queue.cc:237] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:12.344200 6882 raft_consensus.cc:1261] T 1b7d65d849e242b4b6c0bc06422001db P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:12.344177 6948 consensus_peers.cc:485] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef -> Peer 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): Couldn't send request to peer 1d0e738c34af40fa9950c6f086619b3a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b7d65d849e242b4b6c0bc06422001db. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:12.344452 9626 raft_consensus.cc:1261] T 1b7d65d849e242b4b6c0bc06422001db P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.344635 19820 consensus_queue.cc:1035] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:12.344867 20063 consensus_queue.cc:1035] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:12.347292 19820 raft_consensus.cc:2932] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:12.347723 6882 raft_consensus.cc:2932] T 1b7d65d849e242b4b6c0bc06422001db P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:12.351876 6732 catalog_manager.cc:5484] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:12.353307 9626 raft_consensus.cc:2932] T 1b7d65d849e242b4b6c0bc06422001db P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:12.404985 7143 consensus_queue.cc:237] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
W20240503 02:00:12.406919 7080 consensus_peers.cc:485] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 38f126abb17c4654ae3cabe688d513d5. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:12.407194 9755 raft_consensus.cc:1261] T 38f126abb17c4654ae3cabe688d513d5 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.407696 19839 consensus_queue.cc:1035] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:12.407867 7014 raft_consensus.cc:1261] T 38f126abb17c4654ae3cabe688d513d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.408381 19839 consensus_queue.cc:1035] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:12.410285 19839 raft_consensus.cc:2932] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:12.410748 7014 raft_consensus.cc:2932] T 38f126abb17c4654ae3cabe688d513d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:12.414188 6732 catalog_manager.cc:5484] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index -1 to 2, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:12.414644 9755 raft_consensus.cc:2932] T 38f126abb17c4654ae3cabe688d513d5 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:12.420492 20089 ts_tablet_manager.cc:902] T 75175ed25db846b9ba8e8e7b595102ca P 1d0e738c34af40fa9950c6f086619b3a: Initiating tablet copy from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 02:00:12.421244 20089 tablet_copy_client.cc:308] T 75175ed25db846b9ba8e8e7b595102ca P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41979
I20240503 02:00:12.421926 7165 tablet_copy_service.cc:140] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Received BeginTabletCopySession request for tablet 75175ed25db846b9ba8e8e7b595102ca from peer 1d0e738c34af40fa9950c6f086619b3a ({username='slave'} at 127.0.99.132:38863)
I20240503 02:00:12.422040 7165 tablet_copy_service.cc:161] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Beginning new tablet copy session on tablet 75175ed25db846b9ba8e8e7b595102ca from peer 1d0e738c34af40fa9950c6f086619b3a at {username='slave'} at 127.0.99.132:38863: session id = 1d0e738c34af40fa9950c6f086619b3a-75175ed25db846b9ba8e8e7b595102ca
I20240503 02:00:12.422276 20094 ts_tablet_manager.cc:902] T c4930b3ca84749f6a4cd245411f66075 P b7062fbbc2b54fff9e99ec4ea5562c76: Initiating tablet copy from peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:12.422850 20094 tablet_copy_client.cc:308] T c4930b3ca84749f6a4cd245411f66075 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:37387
I20240503 02:00:12.423048 7165 tablet_copy_source_session.cc:205] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:12.423377 9383 tablet_copy_service.cc:140] P 12e8c683db054df3b75c6285b98a5ad8: Received BeginTabletCopySession request for tablet c4930b3ca84749f6a4cd245411f66075 from peer b7062fbbc2b54fff9e99ec4ea5562c76 ({username='slave'} at 127.0.99.135:33951)
I20240503 02:00:12.423481 9383 tablet_copy_service.cc:161] P 12e8c683db054df3b75c6285b98a5ad8: Beginning new tablet copy session on tablet c4930b3ca84749f6a4cd245411f66075 from peer b7062fbbc2b54fff9e99ec4ea5562c76 at {username='slave'} at 127.0.99.135:33951: session id = b7062fbbc2b54fff9e99ec4ea5562c76-c4930b3ca84749f6a4cd245411f66075
I20240503 02:00:12.424115 20089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75175ed25db846b9ba8e8e7b595102ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.424273 20099 ts_tablet_manager.cc:902] T 2084fee295274ad89b407a933fb6070c P bb5364d730cb4489a0bfc6c270b7450a: Initiating tablet copy from peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 02:00:12.424532 9383 tablet_copy_source_session.cc:205] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:12.425487 20099 tablet_copy_client.cc:308] T 2084fee295274ad89b407a933fb6070c P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:42943
I20240503 02:00:12.426641 6903 tablet_copy_service.cc:140] P 5dcc33cc621a4cbbae155f3230fb7c46: Received BeginTabletCopySession request for tablet 2084fee295274ad89b407a933fb6070c from peer bb5364d730cb4489a0bfc6c270b7450a ({username='slave'} at 127.0.99.134:59615)
I20240503 02:00:12.426754 6903 tablet_copy_service.cc:161] P 5dcc33cc621a4cbbae155f3230fb7c46: Beginning new tablet copy session on tablet 2084fee295274ad89b407a933fb6070c from peer bb5364d730cb4489a0bfc6c270b7450a at {username='slave'} at 127.0.99.134:59615: session id = bb5364d730cb4489a0bfc6c270b7450a-2084fee295274ad89b407a933fb6070c
I20240503 02:00:12.426906 20089 tablet_copy_client.cc:783] T 75175ed25db846b9ba8e8e7b595102ca P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:12.427093 20089 tablet_copy_client.cc:647] T 75175ed25db846b9ba8e8e7b595102ca P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:12.427584 20094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4930b3ca84749f6a4cd245411f66075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.427851 6903 tablet_copy_source_session.cc:205] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:12.429591 20089 tablet_copy_client.cc:523] T 75175ed25db846b9ba8e8e7b595102ca P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:12.429698 20099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2084fee295274ad89b407a933fb6070c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.430155 20094 tablet_copy_client.cc:783] T c4930b3ca84749f6a4cd245411f66075 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:12.430310 20094 tablet_copy_client.cc:647] T c4930b3ca84749f6a4cd245411f66075 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:12.431408 20089 tablet_bootstrap.cc:492] T 75175ed25db846b9ba8e8e7b595102ca P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 02:00:12.432464 20099 tablet_copy_client.cc:783] T 2084fee295274ad89b407a933fb6070c P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:12.432629 20099 tablet_copy_client.cc:647] T 2084fee295274ad89b407a933fb6070c P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:12.433452 20094 tablet_copy_client.cc:523] T c4930b3ca84749f6a4cd245411f66075 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:12.435153 20099 tablet_copy_client.cc:523] T 2084fee295274ad89b407a933fb6070c P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:12.435412 20094 tablet_bootstrap.cc:492] T c4930b3ca84749f6a4cd245411f66075 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 02:00:12.438598 20089 tablet_bootstrap.cc:492] T 75175ed25db846b9ba8e8e7b595102ca P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:12.438992 20089 tablet_bootstrap.cc:492] T 75175ed25db846b9ba8e8e7b595102ca P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap complete.
I20240503 02:00:12.439249 20089 ts_tablet_manager.cc:1367] T 75175ed25db846b9ba8e8e7b595102ca P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20240503 02:00:12.439635 20094 tablet_bootstrap.cc:492] T c4930b3ca84749f6a4cd245411f66075 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:12.439946 20094 tablet_bootstrap.cc:492] T c4930b3ca84749f6a4cd245411f66075 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap complete.
I20240503 02:00:12.440135 20094 ts_tablet_manager.cc:1367] T c4930b3ca84749f6a4cd245411f66075 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 02:00:12.440065 20089 raft_consensus.cc:348] T 75175ed25db846b9ba8e8e7b595102ca P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:12.440297 20089 raft_consensus.cc:729] T 75175ed25db846b9ba8e8e7b595102ca P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.436756 20099 tablet_bootstrap.cc:492] T 2084fee295274ad89b407a933fb6070c P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 02:00:12.440548 20089 consensus_queue.cc:260] T 75175ed25db846b9ba8e8e7b595102ca P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:12.440646 20094 raft_consensus.cc:348] T c4930b3ca84749f6a4cd245411f66075 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:12.440769 20094 raft_consensus.cc:729] T c4930b3ca84749f6a4cd245411f66075 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.440913 20094 consensus_queue.cc:260] T c4930b3ca84749f6a4cd245411f66075 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:12.441220 20094 ts_tablet_manager.cc:1398] T c4930b3ca84749f6a4cd245411f66075 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 02:00:12.443064 9383 tablet_copy_service.cc:342] P 12e8c683db054df3b75c6285b98a5ad8: Request end of tablet copy session b7062fbbc2b54fff9e99ec4ea5562c76-c4930b3ca84749f6a4cd245411f66075 received from {username='slave'} at 127.0.99.135:33951
I20240503 02:00:12.443154 9383 tablet_copy_service.cc:433] P 12e8c683db054df3b75c6285b98a5ad8: ending tablet copy session b7062fbbc2b54fff9e99ec4ea5562c76-c4930b3ca84749f6a4cd245411f66075 on tablet c4930b3ca84749f6a4cd245411f66075 with peer b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:12.443598 20089 ts_tablet_manager.cc:1398] T 75175ed25db846b9ba8e8e7b595102ca P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 02:00:12.444730 7165 tablet_copy_service.cc:342] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Request end of tablet copy session 1d0e738c34af40fa9950c6f086619b3a-75175ed25db846b9ba8e8e7b595102ca received from {username='slave'} at 127.0.99.132:38863
I20240503 02:00:12.444818 7165 tablet_copy_service.cc:433] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: ending tablet copy session 1d0e738c34af40fa9950c6f086619b3a-75175ed25db846b9ba8e8e7b595102ca on tablet 75175ed25db846b9ba8e8e7b595102ca with peer 1d0e738c34af40fa9950c6f086619b3a
I20240503 02:00:12.448012 20099 tablet_bootstrap.cc:492] T 2084fee295274ad89b407a933fb6070c P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:12.448436 20099 tablet_bootstrap.cc:492] T 2084fee295274ad89b407a933fb6070c P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap complete.
I20240503 02:00:12.448678 20099 ts_tablet_manager.cc:1367] T 2084fee295274ad89b407a933fb6070c P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20240503 02:00:12.449200 20099 raft_consensus.cc:348] T 2084fee295274ad89b407a933fb6070c P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:12.449331 20099 raft_consensus.cc:729] T 2084fee295274ad89b407a933fb6070c P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.449487 20099 consensus_queue.cc:260] T 2084fee295274ad89b407a933fb6070c P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:12.449826 20099 ts_tablet_manager.cc:1398] T 2084fee295274ad89b407a933fb6070c P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:12.450428 6903 tablet_copy_service.cc:342] P 5dcc33cc621a4cbbae155f3230fb7c46: Request end of tablet copy session bb5364d730cb4489a0bfc6c270b7450a-2084fee295274ad89b407a933fb6070c received from {username='slave'} at 127.0.99.134:59615
I20240503 02:00:12.450552 6903 tablet_copy_service.cc:433] P 5dcc33cc621a4cbbae155f3230fb7c46: ending tablet copy session bb5364d730cb4489a0bfc6c270b7450a-2084fee295274ad89b407a933fb6070c on tablet 2084fee295274ad89b407a933fb6070c with peer bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:12.453686 9362 consensus_queue.cc:237] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:12.455575 7145 raft_consensus.cc:1261] T b886572c6ec9473bbec744cee8de12f5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.455757 9494 raft_consensus.cc:1261] T b886572c6ec9473bbec744cee8de12f5 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:12.456033 9299 consensus_peers.cc:485] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8 -> Peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): Couldn't send request to peer 5dcc33cc621a4cbbae155f3230fb7c46. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b886572c6ec9473bbec744cee8de12f5. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:12.456399 19702 consensus_queue.cc:1035] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:12.456578 18823 consensus_queue.cc:1035] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:12.460258 19702 raft_consensus.cc:2932] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:12.460745 9494 raft_consensus.cc:2932] T b886572c6ec9473bbec744cee8de12f5 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:12.462884 7145 raft_consensus.cc:2932] T b886572c6ec9473bbec744cee8de12f5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:12.467031 6736 catalog_manager.cc:5484] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:12.487311 20132 ts_tablet_manager.cc:902] T 8feff37a8a6043d2b2f7581deb7c7934 P 63331f455be049c1bdd6c23c59ac1bef: Initiating tablet copy from peer db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 02:00:12.488272 20132 tablet_copy_client.cc:308] T 8feff37a8a6043d2b2f7581deb7c7934 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.136:42053
I20240503 02:00:12.488868 9777 tablet_copy_service.cc:140] P db1a0000aaa346f79c0aa4b6c5105191: Received BeginTabletCopySession request for tablet 8feff37a8a6043d2b2f7581deb7c7934 from peer 63331f455be049c1bdd6c23c59ac1bef ({username='slave'} at 127.0.99.130:38135)
I20240503 02:00:12.488999 9777 tablet_copy_service.cc:161] P db1a0000aaa346f79c0aa4b6c5105191: Beginning new tablet copy session on tablet 8feff37a8a6043d2b2f7581deb7c7934 from peer 63331f455be049c1bdd6c23c59ac1bef at {username='slave'} at 127.0.99.130:38135: session id = 63331f455be049c1bdd6c23c59ac1bef-8feff37a8a6043d2b2f7581deb7c7934
I20240503 02:00:12.490090 9777 tablet_copy_source_session.cc:205] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:12.491365 20132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8feff37a8a6043d2b2f7581deb7c7934. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.495249 9494 consensus_queue.cc:237] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:12.497401 9755 raft_consensus.cc:1261] T f20b6a14b8674e6fafefe20ca326d38b P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.497680 9363 raft_consensus.cc:1261] T f20b6a14b8674e6fafefe20ca326d38b P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:12.498258 20143 consensus_queue.cc:1035] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:12.499722 20143 consensus_queue.cc:1035] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:00:12.500985 9429 consensus_peers.cc:485] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a -> Peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): Couldn't send request to peer c688f3df56ac4b6d9b0c30fa4cc8ca2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f20b6a14b8674e6fafefe20ca326d38b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:12.502816 20146 ts_tablet_manager.cc:902] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46: Initiating tablet copy from peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:12.502802 20132 tablet_copy_client.cc:783] T 8feff37a8a6043d2b2f7581deb7c7934 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:12.503017 20132 tablet_copy_client.cc:647] T 8feff37a8a6043d2b2f7581deb7c7934 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:12.503362 20143 raft_consensus.cc:2932] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:12.503662 20146 tablet_copy_client.cc:308] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:37387
I20240503 02:00:12.504086 9383 tablet_copy_service.cc:140] P 12e8c683db054df3b75c6285b98a5ad8: Received BeginTabletCopySession request for tablet beb175f81d9e48229466503ebf7e3c09 from peer 5dcc33cc621a4cbbae155f3230fb7c46 ({username='slave'} at 127.0.99.129:37661)
I20240503 02:00:12.504189 9383 tablet_copy_service.cc:161] P 12e8c683db054df3b75c6285b98a5ad8: Beginning new tablet copy session on tablet beb175f81d9e48229466503ebf7e3c09 from peer 5dcc33cc621a4cbbae155f3230fb7c46 at {username='slave'} at 127.0.99.129:37661: session id = 5dcc33cc621a4cbbae155f3230fb7c46-beb175f81d9e48229466503ebf7e3c09
I20240503 02:00:12.504112 9755 raft_consensus.cc:2932] T f20b6a14b8674e6fafefe20ca326d38b P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:12.505226 9383 tablet_copy_source_session.cc:205] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:12.507810 6735 catalog_manager.cc:5484] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:12.508134 9363 raft_consensus.cc:2932] T f20b6a14b8674e6fafefe20ca326d38b P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:12.510264 20146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beb175f81d9e48229466503ebf7e3c09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.510468 20132 tablet_copy_client.cc:523] T 8feff37a8a6043d2b2f7581deb7c7934 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:12.513397 20146 tablet_copy_client.cc:783] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:12.513576 20146 tablet_copy_client.cc:647] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:12.514664 20132 tablet_bootstrap.cc:492] T 8feff37a8a6043d2b2f7581deb7c7934 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 02:00:12.516162 20146 tablet_copy_client.cc:523] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:12.517702 20146 tablet_bootstrap.cc:492] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 02:00:12.518178 20132 tablet_bootstrap.cc:492] T 8feff37a8a6043d2b2f7581deb7c7934 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:12.518457 20132 tablet_bootstrap.cc:492] T 8feff37a8a6043d2b2f7581deb7c7934 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap complete.
I20240503 02:00:12.518667 20132 ts_tablet_manager.cc:1367] T 8feff37a8a6043d2b2f7581deb7c7934 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 02:00:12.519136 20132 raft_consensus.cc:348] T 8feff37a8a6043d2b2f7581deb7c7934 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:12.519265 20132 raft_consensus.cc:729] T 8feff37a8a6043d2b2f7581deb7c7934 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.519445 20132 consensus_queue.cc:260] T 8feff37a8a6043d2b2f7581deb7c7934 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:12.519853 20132 ts_tablet_manager.cc:1398] T 8feff37a8a6043d2b2f7581deb7c7934 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:12.520289 9777 tablet_copy_service.cc:342] P db1a0000aaa346f79c0aa4b6c5105191: Request end of tablet copy session 63331f455be049c1bdd6c23c59ac1bef-8feff37a8a6043d2b2f7581deb7c7934 received from {username='slave'} at 127.0.99.130:38135
I20240503 02:00:12.520364 9777 tablet_copy_service.cc:433] P db1a0000aaa346f79c0aa4b6c5105191: ending tablet copy session 63331f455be049c1bdd6c23c59ac1bef-8feff37a8a6043d2b2f7581deb7c7934 on tablet 8feff37a8a6043d2b2f7581deb7c7934 with peer 63331f455be049c1bdd6c23c59ac1bef
I20240503 02:00:12.527164 20146 tablet_bootstrap.cc:492] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:12.527448 20146 tablet_bootstrap.cc:492] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap complete.
I20240503 02:00:12.527631 20146 ts_tablet_manager.cc:1367] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240503 02:00:12.528367 20146 raft_consensus.cc:348] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:12.528561 20146 raft_consensus.cc:729] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.528751 20146 consensus_queue.cc:260] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:12.529470 20146 ts_tablet_manager.cc:1398] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:00:12.530247 9383 tablet_copy_service.cc:342] P 12e8c683db054df3b75c6285b98a5ad8: Request end of tablet copy session 5dcc33cc621a4cbbae155f3230fb7c46-beb175f81d9e48229466503ebf7e3c09 received from {username='slave'} at 127.0.99.129:37661
I20240503 02:00:12.530364 9383 tablet_copy_service.cc:433] P 12e8c683db054df3b75c6285b98a5ad8: ending tablet copy session 5dcc33cc621a4cbbae155f3230fb7c46-beb175f81d9e48229466503ebf7e3c09 on tablet beb175f81d9e48229466503ebf7e3c09 with peer 5dcc33cc621a4cbbae155f3230fb7c46
I20240503 02:00:12.554407 20094 ts_tablet_manager.cc:902] T d2fbadb911aa4bd7b4db021860c202d3 P b7062fbbc2b54fff9e99ec4ea5562c76: Initiating tablet copy from peer 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 02:00:12.554924 20094 tablet_copy_client.cc:308] T d2fbadb911aa4bd7b4db021860c202d3 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36271
I20240503 02:00:12.555505 7295 tablet_copy_service.cc:140] P 1d0e738c34af40fa9950c6f086619b3a: Received BeginTabletCopySession request for tablet d2fbadb911aa4bd7b4db021860c202d3 from peer b7062fbbc2b54fff9e99ec4ea5562c76 ({username='slave'} at 127.0.99.135:51921)
I20240503 02:00:12.555624 7295 tablet_copy_service.cc:161] P 1d0e738c34af40fa9950c6f086619b3a: Beginning new tablet copy session on tablet d2fbadb911aa4bd7b4db021860c202d3 from peer b7062fbbc2b54fff9e99ec4ea5562c76 at {username='slave'} at 127.0.99.135:51921: session id = b7062fbbc2b54fff9e99ec4ea5562c76-d2fbadb911aa4bd7b4db021860c202d3
I20240503 02:00:12.556902 7295 tablet_copy_source_session.cc:205] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:12.557847 20094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2fbadb911aa4bd7b4db021860c202d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.560170 20094 tablet_copy_client.cc:783] T d2fbadb911aa4bd7b4db021860c202d3 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:12.560341 20094 tablet_copy_client.cc:647] T d2fbadb911aa4bd7b4db021860c202d3 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:12.562613 20094 tablet_copy_client.cc:523] T d2fbadb911aa4bd7b4db021860c202d3 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:12.563751 20094 tablet_bootstrap.cc:492] T d2fbadb911aa4bd7b4db021860c202d3 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 02:00:12.570156 6882 consensus_queue.cc:237] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:12.571529 20175 ts_tablet_manager.cc:902] T 7672f42acd9446ef85eced07729c197b P 12e8c683db054df3b75c6285b98a5ad8: Initiating tablet copy from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 02:00:12.572091 20175 tablet_copy_client.cc:308] T 7672f42acd9446ef85eced07729c197b P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41979
I20240503 02:00:12.572203 9757 raft_consensus.cc:1261] T dec7e3036d384f5db51f50ba93acbbc6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:12.572317 7274 raft_consensus.cc:1261] T dec7e3036d384f5db51f50ba93acbbc6 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:12.572710 19481 consensus_queue.cc:1035] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:12.572868 19480 consensus_queue.cc:1035] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:12.573017 7165 tablet_copy_service.cc:140] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Received BeginTabletCopySession request for tablet 7672f42acd9446ef85eced07729c197b from peer 12e8c683db054df3b75c6285b98a5ad8 ({username='slave'} at 127.0.99.133:44539)
I20240503 02:00:12.573115 7165 tablet_copy_service.cc:161] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Beginning new tablet copy session on tablet 7672f42acd9446ef85eced07729c197b from peer 12e8c683db054df3b75c6285b98a5ad8 at {username='slave'} at 127.0.99.133:44539: session id = 12e8c683db054df3b75c6285b98a5ad8-7672f42acd9446ef85eced07729c197b
I20240503 02:00:12.574115 7165 tablet_copy_source_session.cc:205] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Tablet Copy: opened 0 blocks and 1 log segments
W20240503 02:00:12.575671 6818 consensus_peers.cc:485] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dec7e3036d384f5db51f50ba93acbbc6. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:12.578841 19481 raft_consensus.cc:2932] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:12.582805 6732 catalog_manager.cc:5484] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index 5 to 6, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:12.583168 20175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7672f42acd9446ef85eced07729c197b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.583492 7274 raft_consensus.cc:2932] T dec7e3036d384f5db51f50ba93acbbc6 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:12.586592 9757 raft_consensus.cc:2932] T dec7e3036d384f5db51f50ba93acbbc6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:12.591625 20175 tablet_copy_client.cc:783] T 7672f42acd9446ef85eced07729c197b P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:12.591961 20175 tablet_copy_client.cc:647] T 7672f42acd9446ef85eced07729c197b P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:12.594108 20175 tablet_copy_client.cc:523] T 7672f42acd9446ef85eced07729c197b P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:12.595674 20175 tablet_bootstrap.cc:492] T 7672f42acd9446ef85eced07729c197b P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 02:00:12.599656 20175 tablet_bootstrap.cc:492] T 7672f42acd9446ef85eced07729c197b P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:12.599927 20175 tablet_bootstrap.cc:492] T 7672f42acd9446ef85eced07729c197b P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap complete.
I20240503 02:00:12.600073 20175 ts_tablet_manager.cc:1367] T 7672f42acd9446ef85eced07729c197b P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240503 02:00:12.600528 20175 raft_consensus.cc:348] T 7672f42acd9446ef85eced07729c197b P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:12.600641 20175 raft_consensus.cc:729] T 7672f42acd9446ef85eced07729c197b P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.600788 20175 consensus_queue.cc:260] T 7672f42acd9446ef85eced07729c197b P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:12.601089 20175 ts_tablet_manager.cc:1398] T 7672f42acd9446ef85eced07729c197b P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:12.601644 7165 tablet_copy_service.cc:342] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Request end of tablet copy session 12e8c683db054df3b75c6285b98a5ad8-7672f42acd9446ef85eced07729c197b received from {username='slave'} at 127.0.99.133:44539
I20240503 02:00:12.601738 7165 tablet_copy_service.cc:433] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: ending tablet copy session 12e8c683db054df3b75c6285b98a5ad8-7672f42acd9446ef85eced07729c197b on tablet 7672f42acd9446ef85eced07729c197b with peer 12e8c683db054df3b75c6285b98a5ad8
I20240503 02:00:12.604609 20094 tablet_bootstrap.cc:492] T d2fbadb911aa4bd7b4db021860c202d3 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:12.604919 20094 tablet_bootstrap.cc:492] T d2fbadb911aa4bd7b4db021860c202d3 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap complete.
I20240503 02:00:12.605111 20094 ts_tablet_manager.cc:1367] T d2fbadb911aa4bd7b4db021860c202d3 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.041s user 0.029s sys 0.000s
I20240503 02:00:12.605527 20094 raft_consensus.cc:348] T d2fbadb911aa4bd7b4db021860c202d3 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:12.605664 20094 raft_consensus.cc:729] T d2fbadb911aa4bd7b4db021860c202d3 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.605819 20094 consensus_queue.cc:260] T d2fbadb911aa4bd7b4db021860c202d3 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:12.607043 20094 ts_tablet_manager.cc:1398] T d2fbadb911aa4bd7b4db021860c202d3 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 02:00:12.607539 7295 tablet_copy_service.cc:342] P 1d0e738c34af40fa9950c6f086619b3a: Request end of tablet copy session b7062fbbc2b54fff9e99ec4ea5562c76-d2fbadb911aa4bd7b4db021860c202d3 received from {username='slave'} at 127.0.99.135:51921
I20240503 02:00:12.607614 7295 tablet_copy_service.cc:433] P 1d0e738c34af40fa9950c6f086619b3a: ending tablet copy session b7062fbbc2b54fff9e99ec4ea5562c76-d2fbadb911aa4bd7b4db021860c202d3 on tablet d2fbadb911aa4bd7b4db021860c202d3 with peer b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:12.626894 20194 ts_tablet_manager.cc:902] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Initiating tablet copy from peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:12.627240 20132 ts_tablet_manager.cc:902] T aab7d8c559ff446192f471aacaa47584 P 63331f455be049c1bdd6c23c59ac1bef: Initiating tablet copy from peer 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 02:00:12.627453 20194 tablet_copy_client.cc:308] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:37387
I20240503 02:00:12.627638 20132 tablet_copy_client.cc:308] T aab7d8c559ff446192f471aacaa47584 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.132:36271
I20240503 02:00:12.628060 9383 tablet_copy_service.cc:140] P 12e8c683db054df3b75c6285b98a5ad8: Received BeginTabletCopySession request for tablet 2755530dc6f04baaaafd5d5214b150af from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a ({username='slave'} at 127.0.99.131:49995)
I20240503 02:00:12.628170 9383 tablet_copy_service.cc:161] P 12e8c683db054df3b75c6285b98a5ad8: Beginning new tablet copy session on tablet 2755530dc6f04baaaafd5d5214b150af from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a at {username='slave'} at 127.0.99.131:49995: session id = c688f3df56ac4b6d9b0c30fa4cc8ca2a-2755530dc6f04baaaafd5d5214b150af
I20240503 02:00:12.629293 9383 tablet_copy_source_session.cc:205] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:12.629662 7295 tablet_copy_service.cc:140] P 1d0e738c34af40fa9950c6f086619b3a: Received BeginTabletCopySession request for tablet aab7d8c559ff446192f471aacaa47584 from peer 63331f455be049c1bdd6c23c59ac1bef ({username='slave'} at 127.0.99.130:43003)
I20240503 02:00:12.629776 7295 tablet_copy_service.cc:161] P 1d0e738c34af40fa9950c6f086619b3a: Beginning new tablet copy session on tablet aab7d8c559ff446192f471aacaa47584 from peer 63331f455be049c1bdd6c23c59ac1bef at {username='slave'} at 127.0.99.130:43003: session id = 63331f455be049c1bdd6c23c59ac1bef-aab7d8c559ff446192f471aacaa47584
I20240503 02:00:12.630223 20194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2755530dc6f04baaaafd5d5214b150af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.630592 7295 tablet_copy_source_session.cc:205] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:12.631419 20132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aab7d8c559ff446192f471aacaa47584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.633069 20194 tablet_copy_client.cc:783] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:12.633265 20194 tablet_copy_client.cc:647] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:12.633742 20132 tablet_copy_client.cc:783] T aab7d8c559ff446192f471aacaa47584 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:12.633891 20132 tablet_copy_client.cc:647] T aab7d8c559ff446192f471aacaa47584 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:12.635507 20132 tablet_copy_client.cc:523] T aab7d8c559ff446192f471aacaa47584 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:12.636314 20194 tablet_copy_client.cc:523] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:12.636857 20132 tablet_bootstrap.cc:492] T aab7d8c559ff446192f471aacaa47584 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 02:00:12.640152 20194 tablet_bootstrap.cc:492] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 02:00:12.641167 20132 tablet_bootstrap.cc:492] T aab7d8c559ff446192f471aacaa47584 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:12.641443 20132 tablet_bootstrap.cc:492] T aab7d8c559ff446192f471aacaa47584 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap complete.
I20240503 02:00:12.641623 20132 ts_tablet_manager.cc:1367] T aab7d8c559ff446192f471aacaa47584 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240503 02:00:12.642141 20132 raft_consensus.cc:348] T aab7d8c559ff446192f471aacaa47584 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:12.642292 20132 raft_consensus.cc:729] T aab7d8c559ff446192f471aacaa47584 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.642453 20132 consensus_queue.cc:260] T aab7d8c559ff446192f471aacaa47584 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:12.643123 20132 ts_tablet_manager.cc:1398] T aab7d8c559ff446192f471aacaa47584 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 02:00:12.643942 7295 tablet_copy_service.cc:342] P 1d0e738c34af40fa9950c6f086619b3a: Request end of tablet copy session 63331f455be049c1bdd6c23c59ac1bef-aab7d8c559ff446192f471aacaa47584 received from {username='slave'} at 127.0.99.130:43003
I20240503 02:00:12.644135 7295 tablet_copy_service.cc:433] P 1d0e738c34af40fa9950c6f086619b3a: ending tablet copy session 63331f455be049c1bdd6c23c59ac1bef-aab7d8c559ff446192f471aacaa47584 on tablet aab7d8c559ff446192f471aacaa47584 with peer 63331f455be049c1bdd6c23c59ac1bef
I20240503 02:00:12.648677 20194 tablet_bootstrap.cc:492] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:12.649245 20194 tablet_bootstrap.cc:492] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap complete.
I20240503 02:00:12.649442 20194 ts_tablet_manager.cc:1367] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20240503 02:00:12.650132 20089 ts_tablet_manager.cc:902] T 6364f3919c42498ca47ba114d525279b P 1d0e738c34af40fa9950c6f086619b3a: Initiating tablet copy from peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 02:00:12.650537 20194 raft_consensus.cc:348] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:12.650668 20089 tablet_copy_client.cc:308] T 6364f3919c42498ca47ba114d525279b P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:42943
I20240503 02:00:12.650694 20194 raft_consensus.cc:729] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.650857 20194 consensus_queue.cc:260] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:12.651142 6903 tablet_copy_service.cc:140] P 5dcc33cc621a4cbbae155f3230fb7c46: Received BeginTabletCopySession request for tablet 6364f3919c42498ca47ba114d525279b from peer 1d0e738c34af40fa9950c6f086619b3a ({username='slave'} at 127.0.99.132:58605)
I20240503 02:00:12.651261 6903 tablet_copy_service.cc:161] P 5dcc33cc621a4cbbae155f3230fb7c46: Beginning new tablet copy session on tablet 6364f3919c42498ca47ba114d525279b from peer 1d0e738c34af40fa9950c6f086619b3a at {username='slave'} at 127.0.99.132:58605: session id = 1d0e738c34af40fa9950c6f086619b3a-6364f3919c42498ca47ba114d525279b
I20240503 02:00:12.652470 6903 tablet_copy_source_session.cc:205] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:12.652694 20194 ts_tablet_manager.cc:1398] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:00:12.654119 9383 tablet_copy_service.cc:342] P 12e8c683db054df3b75c6285b98a5ad8: Request end of tablet copy session c688f3df56ac4b6d9b0c30fa4cc8ca2a-2755530dc6f04baaaafd5d5214b150af received from {username='slave'} at 127.0.99.131:49995
I20240503 02:00:12.654280 9383 tablet_copy_service.cc:433] P 12e8c683db054df3b75c6285b98a5ad8: ending tablet copy session c688f3df56ac4b6d9b0c30fa4cc8ca2a-2755530dc6f04baaaafd5d5214b150af on tablet 2755530dc6f04baaaafd5d5214b150af with peer c688f3df56ac4b6d9b0c30fa4cc8ca2a
I20240503 02:00:12.655788 20089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6364f3919c42498ca47ba114d525279b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.660142 20089 tablet_copy_client.cc:783] T 6364f3919c42498ca47ba114d525279b P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:12.660334 20089 tablet_copy_client.cc:647] T 6364f3919c42498ca47ba114d525279b P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:12.662011 20089 tablet_copy_client.cc:523] T 6364f3919c42498ca47ba114d525279b P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:12.663710 20089 tablet_bootstrap.cc:492] T 6364f3919c42498ca47ba114d525279b P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 02:00:12.668179 20089 tablet_bootstrap.cc:492] T 6364f3919c42498ca47ba114d525279b P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:12.668764 20089 tablet_bootstrap.cc:492] T 6364f3919c42498ca47ba114d525279b P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap complete.
I20240503 02:00:12.669064 20089 ts_tablet_manager.cc:1367] T 6364f3919c42498ca47ba114d525279b P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240503 02:00:12.669847 20089 raft_consensus.cc:348] T 6364f3919c42498ca47ba114d525279b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:12.670017 20089 raft_consensus.cc:729] T 6364f3919c42498ca47ba114d525279b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.670203 20089 consensus_queue.cc:260] T 6364f3919c42498ca47ba114d525279b P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:12.670658 20089 ts_tablet_manager.cc:1398] T 6364f3919c42498ca47ba114d525279b P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20240503 02:00:12.671386 6903 tablet_copy_service.cc:342] P 5dcc33cc621a4cbbae155f3230fb7c46: Request end of tablet copy session 1d0e738c34af40fa9950c6f086619b3a-6364f3919c42498ca47ba114d525279b received from {username='slave'} at 127.0.99.132:58605
I20240503 02:00:12.671480 6903 tablet_copy_service.cc:433] P 5dcc33cc621a4cbbae155f3230fb7c46: ending tablet copy session 1d0e738c34af40fa9950c6f086619b3a-6364f3919c42498ca47ba114d525279b on tablet 6364f3919c42498ca47ba114d525279b with peer 1d0e738c34af40fa9950c6f086619b3a
I20240503 02:00:12.704676 20240 ts_tablet_manager.cc:902] T 8295ea038a88485095544cc608b000c2 P db1a0000aaa346f79c0aa4b6c5105191: Initiating tablet copy from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 02:00:12.705670 20240 tablet_copy_client.cc:308] T 8295ea038a88485095544cc608b000c2 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41979
I20240503 02:00:12.707476 7165 tablet_copy_service.cc:140] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Received BeginTabletCopySession request for tablet 8295ea038a88485095544cc608b000c2 from peer db1a0000aaa346f79c0aa4b6c5105191 ({username='slave'} at 127.0.99.136:40777)
I20240503 02:00:12.707607 7165 tablet_copy_service.cc:161] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Beginning new tablet copy session on tablet 8295ea038a88485095544cc608b000c2 from peer db1a0000aaa346f79c0aa4b6c5105191 at {username='slave'} at 127.0.99.136:40777: session id = db1a0000aaa346f79c0aa4b6c5105191-8295ea038a88485095544cc608b000c2
I20240503 02:00:12.708671 7165 tablet_copy_source_session.cc:205] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:12.715620 20240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8295ea038a88485095544cc608b000c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.720252 20240 tablet_copy_client.cc:783] T 8295ea038a88485095544cc608b000c2 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:12.724452 20240 tablet_copy_client.cc:647] T 8295ea038a88485095544cc608b000c2 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:12.725260 20089 ts_tablet_manager.cc:902] T 1b7d65d849e242b4b6c0bc06422001db P 1d0e738c34af40fa9950c6f086619b3a: Initiating tablet copy from peer 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 02:00:12.725757 20089 tablet_copy_client.cc:308] T 1b7d65d849e242b4b6c0bc06422001db P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:37231
I20240503 02:00:12.726208 7034 tablet_copy_service.cc:140] P 63331f455be049c1bdd6c23c59ac1bef: Received BeginTabletCopySession request for tablet 1b7d65d849e242b4b6c0bc06422001db from peer 1d0e738c34af40fa9950c6f086619b3a ({username='slave'} at 127.0.99.132:48463)
I20240503 02:00:12.726330 7034 tablet_copy_service.cc:161] P 63331f455be049c1bdd6c23c59ac1bef: Beginning new tablet copy session on tablet 1b7d65d849e242b4b6c0bc06422001db from peer 1d0e738c34af40fa9950c6f086619b3a at {username='slave'} at 127.0.99.132:48463: session id = 1d0e738c34af40fa9950c6f086619b3a-1b7d65d849e242b4b6c0bc06422001db
I20240503 02:00:12.727290 7034 tablet_copy_source_session.cc:205] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:12.728283 20245 ts_tablet_manager.cc:902] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191: Initiating tablet copy from peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:12.728863 20245 tablet_copy_client.cc:308] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:46087
I20240503 02:00:12.729022 20089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b7d65d849e242b4b6c0bc06422001db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.729727 9646 tablet_copy_service.cc:140] P b7062fbbc2b54fff9e99ec4ea5562c76: Received BeginTabletCopySession request for tablet 944657ceab07465092a950a8c793ceff from peer db1a0000aaa346f79c0aa4b6c5105191 ({username='slave'} at 127.0.99.136:51467)
I20240503 02:00:12.729858 9646 tablet_copy_service.cc:161] P b7062fbbc2b54fff9e99ec4ea5562c76: Beginning new tablet copy session on tablet 944657ceab07465092a950a8c793ceff from peer db1a0000aaa346f79c0aa4b6c5105191 at {username='slave'} at 127.0.99.136:51467: session id = db1a0000aaa346f79c0aa4b6c5105191-944657ceab07465092a950a8c793ceff
I20240503 02:00:12.730743 9646 tablet_copy_source_session.cc:205] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:12.731591 20245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 944657ceab07465092a950a8c793ceff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.732585 20089 tablet_copy_client.cc:783] T 1b7d65d849e242b4b6c0bc06422001db P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:12.732779 20089 tablet_copy_client.cc:647] T 1b7d65d849e242b4b6c0bc06422001db P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:12.735884 20194 ts_tablet_manager.cc:902] T d4fa0337228d4f8cb6b6c3aab437619e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Initiating tablet copy from peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:12.736372 20194 tablet_copy_client.cc:308] T d4fa0337228d4f8cb6b6c3aab437619e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.135:46087
I20240503 02:00:12.737752 9646 tablet_copy_service.cc:140] P b7062fbbc2b54fff9e99ec4ea5562c76: Received BeginTabletCopySession request for tablet d4fa0337228d4f8cb6b6c3aab437619e from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a ({username='slave'} at 127.0.99.131:33625)
I20240503 02:00:12.737875 9646 tablet_copy_service.cc:161] P b7062fbbc2b54fff9e99ec4ea5562c76: Beginning new tablet copy session on tablet d4fa0337228d4f8cb6b6c3aab437619e from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a at {username='slave'} at 127.0.99.131:33625: session id = c688f3df56ac4b6d9b0c30fa4cc8ca2a-d4fa0337228d4f8cb6b6c3aab437619e
I20240503 02:00:12.738612 20245 tablet_copy_client.cc:783] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:12.738878 9646 tablet_copy_source_session.cc:205] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:12.738915 20245 tablet_copy_client.cc:647] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:12.740116 20194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4fa0337228d4f8cb6b6c3aab437619e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.731595 20240 tablet_copy_client.cc:523] T 8295ea038a88485095544cc608b000c2 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:12.745894 20089 tablet_copy_client.cc:523] T 1b7d65d849e242b4b6c0bc06422001db P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:12.746204 20245 tablet_copy_client.cc:523] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:12.746795 20194 tablet_copy_client.cc:783] T d4fa0337228d4f8cb6b6c3aab437619e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:12.748158 20245 tablet_bootstrap.cc:492] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 02:00:12.748219 20194 tablet_copy_client.cc:647] T d4fa0337228d4f8cb6b6c3aab437619e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:12.749287 20240 tablet_bootstrap.cc:492] T 8295ea038a88485095544cc608b000c2 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 02:00:12.752110 20089 tablet_bootstrap.cc:492] T 1b7d65d849e242b4b6c0bc06422001db P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 02:00:12.752969 20194 tablet_copy_client.cc:523] T d4fa0337228d4f8cb6b6c3aab437619e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:12.754771 20194 tablet_bootstrap.cc:492] T d4fa0337228d4f8cb6b6c3aab437619e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 02:00:12.756868 20089 tablet_bootstrap.cc:492] T 1b7d65d849e242b4b6c0bc06422001db P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:12.757118 20089 tablet_bootstrap.cc:492] T 1b7d65d849e242b4b6c0bc06422001db P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap complete.
I20240503 02:00:12.757277 20089 ts_tablet_manager.cc:1367] T 1b7d65d849e242b4b6c0bc06422001db P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240503 02:00:12.757786 20089 raft_consensus.cc:348] T 1b7d65d849e242b4b6c0bc06422001db P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:12.757905 20089 raft_consensus.cc:729] T 1b7d65d849e242b4b6c0bc06422001db P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.758030 20089 consensus_queue.cc:260] T 1b7d65d849e242b4b6c0bc06422001db P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:12.758374 20089 ts_tablet_manager.cc:1398] T 1b7d65d849e242b4b6c0bc06422001db P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:12.759018 20194 tablet_bootstrap.cc:492] T d4fa0337228d4f8cb6b6c3aab437619e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:12.759279 20194 tablet_bootstrap.cc:492] T d4fa0337228d4f8cb6b6c3aab437619e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap complete.
I20240503 02:00:12.759449 20194 ts_tablet_manager.cc:1367] T d4fa0337228d4f8cb6b6c3aab437619e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:00:12.760471 7034 tablet_copy_service.cc:342] P 63331f455be049c1bdd6c23c59ac1bef: Request end of tablet copy session 1d0e738c34af40fa9950c6f086619b3a-1b7d65d849e242b4b6c0bc06422001db received from {username='slave'} at 127.0.99.132:48463
I20240503 02:00:12.760567 7034 tablet_copy_service.cc:433] P 63331f455be049c1bdd6c23c59ac1bef: ending tablet copy session 1d0e738c34af40fa9950c6f086619b3a-1b7d65d849e242b4b6c0bc06422001db on tablet 1b7d65d849e242b4b6c0bc06422001db with peer 1d0e738c34af40fa9950c6f086619b3a
I20240503 02:00:12.760704 20194 raft_consensus.cc:348] T d4fa0337228d4f8cb6b6c3aab437619e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:12.760855 20194 raft_consensus.cc:729] T d4fa0337228d4f8cb6b6c3aab437619e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.762634 20194 consensus_queue.cc:260] T d4fa0337228d4f8cb6b6c3aab437619e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:12.765211 20194 ts_tablet_manager.cc:1398] T d4fa0337228d4f8cb6b6c3aab437619e P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 02:00:12.765702 9646 tablet_copy_service.cc:342] P b7062fbbc2b54fff9e99ec4ea5562c76: Request end of tablet copy session c688f3df56ac4b6d9b0c30fa4cc8ca2a-d4fa0337228d4f8cb6b6c3aab437619e received from {username='slave'} at 127.0.99.131:33625
I20240503 02:00:12.765781 9646 tablet_copy_service.cc:433] P b7062fbbc2b54fff9e99ec4ea5562c76: ending tablet copy session c688f3df56ac4b6d9b0c30fa4cc8ca2a-d4fa0337228d4f8cb6b6c3aab437619e on tablet d4fa0337228d4f8cb6b6c3aab437619e with peer c688f3df56ac4b6d9b0c30fa4cc8ca2a
I20240503 02:00:12.774904 20245 tablet_bootstrap.cc:492] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:12.775266 20245 tablet_bootstrap.cc:492] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap complete.
I20240503 02:00:12.775470 20245 ts_tablet_manager.cc:1367] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20240503 02:00:12.778182 20245 raft_consensus.cc:348] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:12.778323 20245 raft_consensus.cc:729] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.778501 20245 consensus_queue.cc:260] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:12.778930 20245 ts_tablet_manager.cc:1398] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:00:12.779544 9646 tablet_copy_service.cc:342] P b7062fbbc2b54fff9e99ec4ea5562c76: Request end of tablet copy session db1a0000aaa346f79c0aa4b6c5105191-944657ceab07465092a950a8c793ceff received from {username='slave'} at 127.0.99.136:51467
I20240503 02:00:12.779693 9646 tablet_copy_service.cc:433] P b7062fbbc2b54fff9e99ec4ea5562c76: ending tablet copy session db1a0000aaa346f79c0aa4b6c5105191-944657ceab07465092a950a8c793ceff on tablet 944657ceab07465092a950a8c793ceff with peer db1a0000aaa346f79c0aa4b6c5105191
I20240503 02:00:12.795317 20240 tablet_bootstrap.cc:492] T 8295ea038a88485095544cc608b000c2 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:12.795696 20240 tablet_bootstrap.cc:492] T 8295ea038a88485095544cc608b000c2 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap complete.
I20240503 02:00:12.795922 20240 ts_tablet_manager.cc:1367] T 8295ea038a88485095544cc608b000c2 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.051s user 0.032s sys 0.001s
I20240503 02:00:12.796397 20240 raft_consensus.cc:348] T 8295ea038a88485095544cc608b000c2 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:12.796525 20240 raft_consensus.cc:729] T 8295ea038a88485095544cc608b000c2 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 02:00:12.796669 20240 consensus_queue.cc:260] T 8295ea038a88485095544cc608b000c2 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:12.797127 20240 ts_tablet_manager.cc:1398] T 8295ea038a88485095544cc608b000c2 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 02:00:12.797978 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:12.802083 7165 tablet_copy_service.cc:342] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Request end of tablet copy session db1a0000aaa346f79c0aa4b6c5105191-8295ea038a88485095544cc608b000c2 received from {username='slave'} at 127.0.99.136:40777
I20240503 02:00:12.802196 7165 tablet_copy_service.cc:433] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: ending tablet copy session db1a0000aaa346f79c0aa4b6c5105191-8295ea038a88485095544cc608b000c2 on tablet 8295ea038a88485095544cc608b000c2 with peer db1a0000aaa346f79c0aa4b6c5105191
I20240503 02:00:12.810406 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:12.818902 9494 raft_consensus.cc:1203] T 2084fee295274ad89b407a933fb6070c P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:12.823042 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:12.830288 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:12.832841 6861 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:12.839309 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:12.840531 9474 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:12.853901 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:12.878890 7274 raft_consensus.cc:1203] T 75175ed25db846b9ba8e8e7b595102ca P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:12.879650 9626 raft_consensus.cc:1203] T c4930b3ca84749f6a4cd245411f66075 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:12.916277 7014 raft_consensus.cc:1203] T 8feff37a8a6043d2b2f7581deb7c7934 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:12.930006 6882 raft_consensus.cc:1203] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:12.973060 9363 raft_consensus.cc:1203] T 7672f42acd9446ef85eced07729c197b P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:12.977600 20194 ts_tablet_manager.cc:902] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Initiating tablet copy from peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 02:00:12.978152 20194 tablet_copy_client.cc:308] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:43471
I20240503 02:00:12.978946 9514 tablet_copy_service.cc:140] P bb5364d730cb4489a0bfc6c270b7450a: Received BeginTabletCopySession request for tablet f20b6a14b8674e6fafefe20ca326d38b from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a ({username='slave'} at 127.0.99.131:52471)
I20240503 02:00:12.979066 9514 tablet_copy_service.cc:161] P bb5364d730cb4489a0bfc6c270b7450a: Beginning new tablet copy session on tablet f20b6a14b8674e6fafefe20ca326d38b from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a at {username='slave'} at 127.0.99.131:52471: session id = c688f3df56ac4b6d9b0c30fa4cc8ca2a-f20b6a14b8674e6fafefe20ca326d38b
I20240503 02:00:12.980187 9514 tablet_copy_source_session.cc:205] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:12.982013 20194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f20b6a14b8674e6fafefe20ca326d38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.984395 20194 tablet_copy_client.cc:783] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:12.984583 20194 tablet_copy_client.cc:647] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:12.986186 20094 ts_tablet_manager.cc:902] T dec7e3036d384f5db51f50ba93acbbc6 P b7062fbbc2b54fff9e99ec4ea5562c76: Initiating tablet copy from peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 02:00:12.986745 20094 tablet_copy_client.cc:308] T dec7e3036d384f5db51f50ba93acbbc6 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:42943
I20240503 02:00:12.987246 6903 tablet_copy_service.cc:140] P 5dcc33cc621a4cbbae155f3230fb7c46: Received BeginTabletCopySession request for tablet dec7e3036d384f5db51f50ba93acbbc6 from peer b7062fbbc2b54fff9e99ec4ea5562c76 ({username='slave'} at 127.0.99.135:54599)
I20240503 02:00:12.987354 6903 tablet_copy_service.cc:161] P 5dcc33cc621a4cbbae155f3230fb7c46: Beginning new tablet copy session on tablet dec7e3036d384f5db51f50ba93acbbc6 from peer b7062fbbc2b54fff9e99ec4ea5562c76 at {username='slave'} at 127.0.99.135:54599: session id = b7062fbbc2b54fff9e99ec4ea5562c76-dec7e3036d384f5db51f50ba93acbbc6
I20240503 02:00:12.988281 6903 tablet_copy_source_session.cc:205] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:12.989626 20094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dec7e3036d384f5db51f50ba93acbbc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:12.991937 20094 tablet_copy_client.cc:783] T dec7e3036d384f5db51f50ba93acbbc6 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:12.992117 20094 tablet_copy_client.cc:647] T dec7e3036d384f5db51f50ba93acbbc6 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:12.992931 20194 tablet_copy_client.cc:523] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:12.994078 20094 tablet_copy_client.cc:523] T dec7e3036d384f5db51f50ba93acbbc6 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:12.994787 20194 tablet_bootstrap.cc:492] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 02:00:12.996284 20094 tablet_bootstrap.cc:492] T dec7e3036d384f5db51f50ba93acbbc6 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 02:00:13.001168 20194 tablet_bootstrap.cc:492] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:13.001516 20194 tablet_bootstrap.cc:492] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap complete.
I20240503 02:00:13.001705 20194 ts_tablet_manager.cc:1367] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 02:00:13.002241 20194 raft_consensus.cc:348] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:13.002960 20194 raft_consensus.cc:729] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.003132 20194 consensus_queue.cc:260] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:13.003521 20194 ts_tablet_manager.cc:1398] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:13.004158 9514 tablet_copy_service.cc:342] P bb5364d730cb4489a0bfc6c270b7450a: Request end of tablet copy session c688f3df56ac4b6d9b0c30fa4cc8ca2a-f20b6a14b8674e6fafefe20ca326d38b received from {username='slave'} at 127.0.99.131:52471
I20240503 02:00:13.004244 9514 tablet_copy_service.cc:433] P bb5364d730cb4489a0bfc6c270b7450a: ending tablet copy session c688f3df56ac4b6d9b0c30fa4cc8ca2a-f20b6a14b8674e6fafefe20ca326d38b on tablet f20b6a14b8674e6fafefe20ca326d38b with peer c688f3df56ac4b6d9b0c30fa4cc8ca2a
I20240503 02:00:13.014228 20175 ts_tablet_manager.cc:902] T 38f126abb17c4654ae3cabe688d513d5 P 12e8c683db054df3b75c6285b98a5ad8: Initiating tablet copy from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 02:00:13.014784 20175 tablet_copy_client.cc:308] T 38f126abb17c4654ae3cabe688d513d5 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41979
I20240503 02:00:13.015285 7165 tablet_copy_service.cc:140] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Received BeginTabletCopySession request for tablet 38f126abb17c4654ae3cabe688d513d5 from peer 12e8c683db054df3b75c6285b98a5ad8 ({username='slave'} at 127.0.99.133:44539)
I20240503 02:00:13.015415 7165 tablet_copy_service.cc:161] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Beginning new tablet copy session on tablet 38f126abb17c4654ae3cabe688d513d5 from peer 12e8c683db054df3b75c6285b98a5ad8 at {username='slave'} at 127.0.99.133:44539: session id = 12e8c683db054df3b75c6285b98a5ad8-38f126abb17c4654ae3cabe688d513d5
I20240503 02:00:13.016458 7165 tablet_copy_source_session.cc:205] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:13.017814 20175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38f126abb17c4654ae3cabe688d513d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.020308 20175 tablet_copy_client.cc:783] T 38f126abb17c4654ae3cabe688d513d5 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:13.020493 20175 tablet_copy_client.cc:647] T 38f126abb17c4654ae3cabe688d513d5 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:13.028193 20175 tablet_copy_client.cc:523] T 38f126abb17c4654ae3cabe688d513d5 P 12e8c683db054df3b75c6285b98a5ad8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:13.030041 20175 tablet_bootstrap.cc:492] T 38f126abb17c4654ae3cabe688d513d5 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap starting.
I20240503 02:00:13.034421 20094 tablet_bootstrap.cc:492] T dec7e3036d384f5db51f50ba93acbbc6 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:13.034752 20094 tablet_bootstrap.cc:492] T dec7e3036d384f5db51f50ba93acbbc6 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap complete.
I20240503 02:00:13.034966 20094 ts_tablet_manager.cc:1367] T dec7e3036d384f5db51f50ba93acbbc6 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.039s user 0.024s sys 0.003s
I20240503 02:00:13.035471 20094 raft_consensus.cc:348] T dec7e3036d384f5db51f50ba93acbbc6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:13.035754 20094 raft_consensus.cc:729] T dec7e3036d384f5db51f50ba93acbbc6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.036074 20094 consensus_queue.cc:260] T dec7e3036d384f5db51f50ba93acbbc6 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:13.036569 20094 ts_tablet_manager.cc:1398] T dec7e3036d384f5db51f50ba93acbbc6 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:13.037336 6903 tablet_copy_service.cc:342] P 5dcc33cc621a4cbbae155f3230fb7c46: Request end of tablet copy session b7062fbbc2b54fff9e99ec4ea5562c76-dec7e3036d384f5db51f50ba93acbbc6 received from {username='slave'} at 127.0.99.135:54599
I20240503 02:00:13.037425 6903 tablet_copy_service.cc:433] P 5dcc33cc621a4cbbae155f3230fb7c46: ending tablet copy session b7062fbbc2b54fff9e99ec4ea5562c76-dec7e3036d384f5db51f50ba93acbbc6 on tablet dec7e3036d384f5db51f50ba93acbbc6 with peer b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:13.040854 20175 tablet_bootstrap.cc:492] T 38f126abb17c4654ae3cabe688d513d5 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:13.041188 20175 tablet_bootstrap.cc:492] T 38f126abb17c4654ae3cabe688d513d5 P 12e8c683db054df3b75c6285b98a5ad8: Bootstrap complete.
I20240503 02:00:13.041396 20175 ts_tablet_manager.cc:1367] T 38f126abb17c4654ae3cabe688d513d5 P 12e8c683db054df3b75c6285b98a5ad8: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240503 02:00:13.041931 20175 raft_consensus.cc:348] T 38f126abb17c4654ae3cabe688d513d5 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:13.042079 20175 raft_consensus.cc:729] T 38f126abb17c4654ae3cabe688d513d5 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e8c683db054df3b75c6285b98a5ad8, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.042258 20175 consensus_queue.cc:260] T 38f126abb17c4654ae3cabe688d513d5 P 12e8c683db054df3b75c6285b98a5ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:13.042699 20175 ts_tablet_manager.cc:1398] T 38f126abb17c4654ae3cabe688d513d5 P 12e8c683db054df3b75c6285b98a5ad8: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20240503 02:00:13.043437 7165 tablet_copy_service.cc:342] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Request end of tablet copy session 12e8c683db054df3b75c6285b98a5ad8-38f126abb17c4654ae3cabe688d513d5 received from {username='slave'} at 127.0.99.133:44539
I20240503 02:00:13.043649 7165 tablet_copy_service.cc:433] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: ending tablet copy session 12e8c683db054df3b75c6285b98a5ad8-38f126abb17c4654ae3cabe688d513d5 on tablet 38f126abb17c4654ae3cabe688d513d5 with peer 12e8c683db054df3b75c6285b98a5ad8
I20240503 02:00:13.103571 20337 ts_tablet_manager.cc:902] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46: Initiating tablet copy from peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:13.105350 20337 tablet_copy_client.cc:308] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:37387
I20240503 02:00:13.105959 9383 tablet_copy_service.cc:140] P 12e8c683db054df3b75c6285b98a5ad8: Received BeginTabletCopySession request for tablet b886572c6ec9473bbec744cee8de12f5 from peer 5dcc33cc621a4cbbae155f3230fb7c46 ({username='slave'} at 127.0.99.129:37661)
I20240503 02:00:13.106088 9383 tablet_copy_service.cc:161] P 12e8c683db054df3b75c6285b98a5ad8: Beginning new tablet copy session on tablet b886572c6ec9473bbec744cee8de12f5 from peer 5dcc33cc621a4cbbae155f3230fb7c46 at {username='slave'} at 127.0.99.129:37661: session id = 5dcc33cc621a4cbbae155f3230fb7c46-b886572c6ec9473bbec744cee8de12f5
I20240503 02:00:13.107197 9383 tablet_copy_source_session.cc:205] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:13.108151 20337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b886572c6ec9473bbec744cee8de12f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:13.116294 20337 tablet_copy_client.cc:783] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:13.116542 20337 tablet_copy_client.cc:647] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:13.117411 9757 raft_consensus.cc:1203] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:13.122247 20337 tablet_copy_client.cc:523] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:13.128410 20337 tablet_bootstrap.cc:492] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 02:00:13.129503 9625 raft_consensus.cc:1203] T d2fbadb911aa4bd7b4db021860c202d3 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 02:00:13.138056 20337 tablet_bootstrap.cc:492] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:13.138365 20337 tablet_bootstrap.cc:492] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap complete.
I20240503 02:00:13.139233 20337 ts_tablet_manager.cc:1367] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240503 02:00:13.140308 20337 raft_consensus.cc:348] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:13.140707 20337 raft_consensus.cc:729] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 02:00:13.140908 20337 consensus_queue.cc:260] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:13.141417 20337 ts_tablet_manager.cc:1398] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:13.142143 9383 tablet_copy_service.cc:342] P 12e8c683db054df3b75c6285b98a5ad8: Request end of tablet copy session 5dcc33cc621a4cbbae155f3230fb7c46-b886572c6ec9473bbec744cee8de12f5 received from {username='slave'} at 127.0.99.129:37661
I20240503 02:00:13.142238 9383 tablet_copy_service.cc:433] P 12e8c683db054df3b75c6285b98a5ad8: ending tablet copy session 5dcc33cc621a4cbbae155f3230fb7c46-b886572c6ec9473bbec744cee8de12f5 on tablet b886572c6ec9473bbec744cee8de12f5 with peer 5dcc33cc621a4cbbae155f3230fb7c46
I20240503 02:00:13.169369 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:13.176365 9474 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:13.178859 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:13.179131 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:13.182554 7145 raft_consensus.cc:1203] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:13.187487 9757 raft_consensus.cc:1203] T 8295ea038a88485095544cc608b000c2 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 02:00:13.189690 6861 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:13.195762 7274 raft_consensus.cc:1203] T 6364f3919c42498ca47ba114d525279b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:13.197171 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:13.220906 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:13.230110 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:13.244180 7014 raft_consensus.cc:1203] T aab7d8c559ff446192f471aacaa47584 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:13.250818 7274 raft_consensus.cc:1203] T 1b7d65d849e242b4b6c0bc06422001db P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:13.263934 7145 raft_consensus.cc:1203] T d4fa0337228d4f8cb6b6c3aab437619e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:13.449860 9363 raft_consensus.cc:1203] T 38f126abb17c4654ae3cabe688d513d5 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:13.479527 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:13.487982 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:13.495901 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:13.512248 6861 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:13.517203 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:13.522621 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:13.527982 9474 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:13.531714 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:13.563552 7145 raft_consensus.cc:1203] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:13.578091 9626 raft_consensus.cc:1203] T dec7e3036d384f5db51f50ba93acbbc6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240503 02:00:13.659550 6883 raft_consensus.cc:1203] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:13.803964 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:13.806617 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:13.807451 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:13.815362 6861 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:13.826275 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:13.837602 9474 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:13.846706 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:13.869107 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:13.936233 9626 consensus_queue.cc:237] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:13.937650 7143 raft_consensus.cc:1261] T d4fa0337228d4f8cb6b6c3aab437619e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer b7062fbbc2b54fff9e99ec4ea5562c76: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:13.937691 7012 raft_consensus.cc:1261] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer b7062fbbc2b54fff9e99ec4ea5562c76: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:13.938138 18142 consensus_queue.cc:1035] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:13.938598 20025 consensus_queue.cc:1035] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:13.940268 18142 raft_consensus.cc:2932] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:13.940868 7012 raft_consensus.cc:2932] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:13.940985 7143 raft_consensus.cc:2932] T d4fa0337228d4f8cb6b6c3aab437619e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:13.945241 6741 catalog_manager.cc:5484] T d4fa0337228d4f8cb6b6c3aab437619e P b7062fbbc2b54fff9e99ec4ea5562c76 reported cstate change: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:13.950400 9474 tablet_service.cc:1514] Processing DeleteTablet for tablet d4fa0337228d4f8cb6b6c3aab437619e with delete_type TABLET_DATA_TOMBSTONED (TS bb5364d730cb4489a0bfc6c270b7450a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54678
I20240503 02:00:13.951259 20488 tablet_replica.cc:330] T d4fa0337228d4f8cb6b6c3aab437619e P bb5364d730cb4489a0bfc6c270b7450a: stopping tablet replica
I20240503 02:00:13.951483 20488 raft_consensus.cc:2229] T d4fa0337228d4f8cb6b6c3aab437619e P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:13.951686 20488 raft_consensus.cc:2258] T d4fa0337228d4f8cb6b6c3aab437619e P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:13.953804 20488 ts_tablet_manager.cc:1875] T d4fa0337228d4f8cb6b6c3aab437619e P bb5364d730cb4489a0bfc6c270b7450a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:13.957309 20488 ts_tablet_manager.cc:1888] T d4fa0337228d4f8cb6b6c3aab437619e P bb5364d730cb4489a0bfc6c270b7450a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:00:13.957516 20488 log.cc:1196] T d4fa0337228d4f8cb6b6c3aab437619e P bb5364d730cb4489a0bfc6c270b7450a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/d4fa0337228d4f8cb6b6c3aab437619e
I20240503 02:00:13.958459 6725 catalog_manager.cc:4830] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): tablet d4fa0337228d4f8cb6b6c3aab437619e (table rebalancer_extra_table_40 [id=28d2e532bc114d04b0b1e87094387600]) successfully deleted
I20240503 02:00:14.136353 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:14.143173 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:14.143311 6861 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:14.144443 9474 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:14.174260 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:14.185072 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:14.186970 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:14.193883 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:14.279937 7275 consensus_queue.cc:237] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:14.281234 7012 raft_consensus.cc:1261] T aab7d8c559ff446192f471aacaa47584 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:14.281555 6882 raft_consensus.cc:1261] T aab7d8c559ff446192f471aacaa47584 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:14.281634 19894 consensus_queue.cc:1035] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:14.281967 19894 consensus_queue.cc:1035] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:14.283483 20275 raft_consensus.cc:2932] T aab7d8c559ff446192f471aacaa47584 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:14.284037 7012 raft_consensus.cc:2932] T aab7d8c559ff446192f471aacaa47584 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:14.284255 6882 raft_consensus.cc:2932] T aab7d8c559ff446192f471aacaa47584 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:14.288728 6735 catalog_manager.cc:5484] T aab7d8c559ff446192f471aacaa47584 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:14.292596 9341 tablet_service.cc:1514] Processing DeleteTablet for tablet aab7d8c559ff446192f471aacaa47584 with delete_type TABLET_DATA_TOMBSTONED (TS 12e8c683db054df3b75c6285b98a5ad8 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:36040
I20240503 02:00:14.293162 20558 tablet_replica.cc:330] T aab7d8c559ff446192f471aacaa47584 P 12e8c683db054df3b75c6285b98a5ad8: stopping tablet replica
I20240503 02:00:14.293571 20558 raft_consensus.cc:2229] T aab7d8c559ff446192f471aacaa47584 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:14.293771 20558 raft_consensus.cc:2258] T aab7d8c559ff446192f471aacaa47584 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:14.294257 20558 ts_tablet_manager.cc:1875] T aab7d8c559ff446192f471aacaa47584 P 12e8c683db054df3b75c6285b98a5ad8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:14.297204 20558 ts_tablet_manager.cc:1888] T aab7d8c559ff446192f471aacaa47584 P 12e8c683db054df3b75c6285b98a5ad8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:00:14.297297 20558 log.cc:1196] T aab7d8c559ff446192f471aacaa47584 P 12e8c683db054df3b75c6285b98a5ad8: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/aab7d8c559ff446192f471aacaa47584
I20240503 02:00:14.297945 6727 catalog_manager.cc:4830] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): tablet aab7d8c559ff446192f471aacaa47584 (table rebalancer_extra_table_16 [id=a60aca1dbfe344479254879037f209f3]) successfully deleted
I20240503 02:00:14.456146 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:14.462898 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:14.464391 6861 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:14.464736 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:14.466182 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:14.468771 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:14.482721 9474 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:14.492331 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:14.587800 6882 consensus_queue.cc:237] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:14.589190 9363 raft_consensus.cc:1261] T 6364f3919c42498ca47ba114d525279b P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:14.589186 7275 raft_consensus.cc:1261] T 6364f3919c42498ca47ba114d525279b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:14.589614 20428 consensus_queue.cc:1035] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:14.589830 19934 consensus_queue.cc:1035] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:14.592083 20428 raft_consensus.cc:2932] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:14.592341 9363 raft_consensus.cc:2932] T 6364f3919c42498ca47ba114d525279b P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:14.592455 7275 raft_consensus.cc:2932] T 6364f3919c42498ca47ba114d525279b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:14.596514 6741 catalog_manager.cc:5484] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:14.600973 9605 tablet_service.cc:1514] Processing DeleteTablet for tablet 6364f3919c42498ca47ba114d525279b with delete_type TABLET_DATA_TOMBSTONED (TS b7062fbbc2b54fff9e99ec4ea5562c76 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:58562
I20240503 02:00:14.601373 20631 tablet_replica.cc:330] T 6364f3919c42498ca47ba114d525279b P b7062fbbc2b54fff9e99ec4ea5562c76: stopping tablet replica
I20240503 02:00:14.601519 20631 raft_consensus.cc:2229] T 6364f3919c42498ca47ba114d525279b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:14.601648 20631 raft_consensus.cc:2258] T 6364f3919c42498ca47ba114d525279b P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:14.602146 20631 ts_tablet_manager.cc:1875] T 6364f3919c42498ca47ba114d525279b P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:14.605065 20631 ts_tablet_manager.cc:1888] T 6364f3919c42498ca47ba114d525279b P b7062fbbc2b54fff9e99ec4ea5562c76: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:00:14.605293 20631 log.cc:1196] T 6364f3919c42498ca47ba114d525279b P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/6364f3919c42498ca47ba114d525279b
I20240503 02:00:14.606492 6727 catalog_manager.cc:4830] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): tablet 6364f3919c42498ca47ba114d525279b (table rebalancer_extra_table_38 [id=2fc3f2f8c5d549c8b1f8c504355aede7]) successfully deleted
I20240503 02:00:14.755204 6861 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:14.762475 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:14.766180 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:14.766685 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:14.771184 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:14.774645 9474 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:14.796103 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:14.810794 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:14.874733 9363 consensus_queue.cc:237] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:14.875903 9625 raft_consensus.cc:1261] T 2755530dc6f04baaaafd5d5214b150af P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:14.875972 7143 raft_consensus.cc:1261] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:14.876231 19817 consensus_queue.cc:1035] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:14.876447 18823 consensus_queue.cc:1035] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:14.878221 19817 raft_consensus.cc:2932] T 2755530dc6f04baaaafd5d5214b150af P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:14.878680 7143 raft_consensus.cc:2932] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:14.879029 9625 raft_consensus.cc:2932] T 2755530dc6f04baaaafd5d5214b150af P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:14.882202 6734 catalog_manager.cc:5484] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index 2 to 3, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:14.885526 6992 tablet_service.cc:1514] Processing DeleteTablet for tablet 2755530dc6f04baaaafd5d5214b150af with delete_type TABLET_DATA_TOMBSTONED (TS 63331f455be049c1bdd6c23c59ac1bef not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:35240
I20240503 02:00:14.886646 20703 tablet_replica.cc:330] T 2755530dc6f04baaaafd5d5214b150af P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 02:00:14.886837 20703 raft_consensus.cc:2229] T 2755530dc6f04baaaafd5d5214b150af P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:14.887077 20703 raft_consensus.cc:2258] T 2755530dc6f04baaaafd5d5214b150af P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:14.887753 20703 ts_tablet_manager.cc:1875] T 2755530dc6f04baaaafd5d5214b150af P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:14.892596 20703 ts_tablet_manager.cc:1888] T 2755530dc6f04baaaafd5d5214b150af P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:00:14.892791 20703 log.cc:1196] T 2755530dc6f04baaaafd5d5214b150af P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/2755530dc6f04baaaafd5d5214b150af
I20240503 02:00:14.894898 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet 2755530dc6f04baaaafd5d5214b150af (table rebalancer_extra_table_14 [id=607fa4fc037b4c5ba04342cb6d7fd017]) successfully deleted
I20240503 02:00:15.033823 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:15.037971 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:15.047168 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:15.055181 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:15.058274 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:15.068349 6861 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:15.071331 9474 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:15.071635 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:15.173209 6883 consensus_queue.cc:237] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:15.174592 9756 raft_consensus.cc:1261] T dec7e3036d384f5db51f50ba93acbbc6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:15.174621 9625 raft_consensus.cc:1261] T dec7e3036d384f5db51f50ba93acbbc6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:15.175011 20626 consensus_queue.cc:1035] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:15.175338 20428 consensus_queue.cc:1035] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:15.176946 20428 raft_consensus.cc:2932] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:15.177250 9756 raft_consensus.cc:2932] T dec7e3036d384f5db51f50ba93acbbc6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:15.177567 9626 raft_consensus.cc:2932] T dec7e3036d384f5db51f50ba93acbbc6 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:15.180938 6735 catalog_manager.cc:5484] T dec7e3036d384f5db51f50ba93acbbc6 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: config changed from index 6 to 7, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:15.184599 7254 tablet_service.cc:1514] Processing DeleteTablet for tablet dec7e3036d384f5db51f50ba93acbbc6 with delete_type TABLET_DATA_TOMBSTONED (TS 1d0e738c34af40fa9950c6f086619b3a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45608
I20240503 02:00:15.184989 20772 tablet_replica.cc:330] T dec7e3036d384f5db51f50ba93acbbc6 P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 02:00:15.185132 20772 raft_consensus.cc:2229] T dec7e3036d384f5db51f50ba93acbbc6 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:15.185323 20772 raft_consensus.cc:2258] T dec7e3036d384f5db51f50ba93acbbc6 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:15.186093 20772 ts_tablet_manager.cc:1875] T dec7e3036d384f5db51f50ba93acbbc6 P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:15.191377 20772 ts_tablet_manager.cc:1888] T dec7e3036d384f5db51f50ba93acbbc6 P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:00:15.191494 20772 log.cc:1196] T dec7e3036d384f5db51f50ba93acbbc6 P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/dec7e3036d384f5db51f50ba93acbbc6
I20240503 02:00:15.192169 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet dec7e3036d384f5db51f50ba93acbbc6 (table rebalance_test_table_17 [id=b061a61957e74150a4af26c16c277330]) successfully deleted
I20240503 02:00:15.338585 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:15.355545 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:15.362380 6861 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:15.364490 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:15.370822 9474 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:15.373610 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:15.379312 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:15.389665 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:15.484648 6883 consensus_queue.cc:237] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:15.486112 9493 raft_consensus.cc:1261] T 2084fee295274ad89b407a933fb6070c P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:15.486131 9755 raft_consensus.cc:1261] T 2084fee295274ad89b407a933fb6070c P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:15.486577 20769 consensus_queue.cc:1035] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:15.487030 20626 consensus_queue.cc:1035] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:15.489782 20626 raft_consensus.cc:2932] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 02:00:15.494457 6734 catalog_manager.cc:5484] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index 2 to 3, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 02:00:15.497226 9755 raft_consensus.cc:2932] T 2084fee295274ad89b407a933fb6070c P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 02:00:15.500218 7254 tablet_service.cc:1514] Processing DeleteTablet for tablet 2084fee295274ad89b407a933fb6070c with delete_type TABLET_DATA_TOMBSTONED (TS 1d0e738c34af40fa9950c6f086619b3a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45608
I20240503 02:00:15.500476 20772 tablet_replica.cc:330] T 2084fee295274ad89b407a933fb6070c P 1d0e738c34af40fa9950c6f086619b3a: stopping tablet replica
I20240503 02:00:15.500627 20772 raft_consensus.cc:2229] T 2084fee295274ad89b407a933fb6070c P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:15.500734 20772 raft_consensus.cc:2258] T 2084fee295274ad89b407a933fb6070c P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:15.501134 20772 ts_tablet_manager.cc:1875] T 2084fee295274ad89b407a933fb6070c P 1d0e738c34af40fa9950c6f086619b3a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:15.503643 20772 ts_tablet_manager.cc:1888] T 2084fee295274ad89b407a933fb6070c P 1d0e738c34af40fa9950c6f086619b3a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:00:15.503734 20772 log.cc:1196] T 2084fee295274ad89b407a933fb6070c P 1d0e738c34af40fa9950c6f086619b3a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/wals/2084fee295274ad89b407a933fb6070c
I20240503 02:00:15.503841 9494 raft_consensus.cc:2932] T 2084fee295274ad89b407a933fb6070c P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 02:00:15.506381 6726 catalog_manager.cc:4830] TS 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): tablet 2084fee295274ad89b407a933fb6070c (table rebalancer_extra_table_52 [id=7084fdc09131424f92874f08cc5ef36b]) successfully deleted
I20240503 02:00:15.696949 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:15.698621 6861 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:15.702085 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:15.702137 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:15.702680 9474 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:15.714057 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:15.732970 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:15.737120 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:15.836467 7143 consensus_queue.cc:237] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:15.838655 7014 raft_consensus.cc:1261] T 7672f42acd9446ef85eced07729c197b P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:15.838753 9363 raft_consensus.cc:1261] T 7672f42acd9446ef85eced07729c197b P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:15.839339 20844 consensus_queue.cc:1035] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:15.839536 19683 consensus_queue.cc:1035] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:15.841893 19683 raft_consensus.cc:2932] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:15.842365 7014 raft_consensus.cc:2932] T 7672f42acd9446ef85eced07729c197b P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:15.843271 9363 raft_consensus.cc:2932] T 7672f42acd9446ef85eced07729c197b P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:15.846745 6740 catalog_manager.cc:5484] T 7672f42acd9446ef85eced07729c197b P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index 2 to 3, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:15.851568 9735 tablet_service.cc:1514] Processing DeleteTablet for tablet 7672f42acd9446ef85eced07729c197b with delete_type TABLET_DATA_TOMBSTONED (TS db1a0000aaa346f79c0aa4b6c5105191 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:48152
I20240503 02:00:15.854629 20916 tablet_replica.cc:330] T 7672f42acd9446ef85eced07729c197b P db1a0000aaa346f79c0aa4b6c5105191: stopping tablet replica
I20240503 02:00:15.854995 20916 raft_consensus.cc:2229] T 7672f42acd9446ef85eced07729c197b P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:15.855264 20916 raft_consensus.cc:2258] T 7672f42acd9446ef85eced07729c197b P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:15.856184 20916 ts_tablet_manager.cc:1875] T 7672f42acd9446ef85eced07729c197b P db1a0000aaa346f79c0aa4b6c5105191: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:15.859438 20916 ts_tablet_manager.cc:1888] T 7672f42acd9446ef85eced07729c197b P db1a0000aaa346f79c0aa4b6c5105191: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:00:15.859625 20916 log.cc:1196] T 7672f42acd9446ef85eced07729c197b P db1a0000aaa346f79c0aa4b6c5105191: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/7672f42acd9446ef85eced07729c197b
I20240503 02:00:15.860273 6726 catalog_manager.cc:4830] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): tablet 7672f42acd9446ef85eced07729c197b (table rebalancer_extra_table_22 [id=cad604d7e464473fa089f248c9623507]) successfully deleted
I20240503 02:00:16.027518 6861 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:16.028401 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:16.028738 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:16.051532 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:16.052162 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:16.066149 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:16.075667 9474 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:16.076169 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:16.186790 7275 consensus_queue.cc:237] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:16.188668 9756 raft_consensus.cc:1261] T d2fbadb911aa4bd7b4db021860c202d3 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:16.188994 9626 raft_consensus.cc:1261] T d2fbadb911aa4bd7b4db021860c202d3 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:16.189136 19894 consensus_queue.cc:1035] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:16.189576 19894 consensus_queue.cc:1035] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:16.192103 20988 raft_consensus.cc:2932] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:16.192684 9756 raft_consensus.cc:2932] T d2fbadb911aa4bd7b4db021860c202d3 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:16.192905 9626 raft_consensus.cc:2932] T d2fbadb911aa4bd7b4db021860c202d3 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:16.197036 6736 catalog_manager.cc:5484] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: config changed from index 6 to 7, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:16.200409 7123 tablet_service.cc:1514] Processing DeleteTablet for tablet d2fbadb911aa4bd7b4db021860c202d3 with delete_type TABLET_DATA_TOMBSTONED (TS c688f3df56ac4b6d9b0c30fa4cc8ca2a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50370
I20240503 02:00:16.201109 20990 tablet_replica.cc:330] T d2fbadb911aa4bd7b4db021860c202d3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 02:00:16.201382 20990 raft_consensus.cc:2229] T d2fbadb911aa4bd7b4db021860c202d3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:16.201639 20990 raft_consensus.cc:2258] T d2fbadb911aa4bd7b4db021860c202d3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:16.202425 20990 ts_tablet_manager.cc:1875] T d2fbadb911aa4bd7b4db021860c202d3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:16.209204 20990 ts_tablet_manager.cc:1888] T d2fbadb911aa4bd7b4db021860c202d3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:00:16.209371 20990 log.cc:1196] T d2fbadb911aa4bd7b4db021860c202d3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/d2fbadb911aa4bd7b4db021860c202d3
I20240503 02:00:16.210741 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet d2fbadb911aa4bd7b4db021860c202d3 (table rebalance_test_table_5 [id=eca57da7c1764bd9a55819692641dac8]) successfully deleted
I20240503 02:00:16.380317 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:16.386471 9474 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:16.389734 6861 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:16.391603 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:16.402295 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:16.411839 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:16.419836 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:16.426334 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:16.534736 7143 consensus_queue.cc:237] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:16.536459 9755 raft_consensus.cc:1261] T 8295ea038a88485095544cc608b000c2 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:16.536583 9625 raft_consensus.cc:1261] T 8295ea038a88485095544cc608b000c2 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:00:16.536952 19683 consensus_queue.cc:1035] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:16.537112 20844 consensus_queue.cc:1035] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:16.539485 21061 raft_consensus.cc:2932] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:16.540047 9625 raft_consensus.cc:2932] T 8295ea038a88485095544cc608b000c2 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:16.540117 9755 raft_consensus.cc:2932] T 8295ea038a88485095544cc608b000c2 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:16.543999 6732 catalog_manager.cc:5484] T 8295ea038a88485095544cc608b000c2 P b7062fbbc2b54fff9e99ec4ea5562c76 reported cstate change: config changed from index 6 to 7, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:16.548259 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet 8295ea038a88485095544cc608b000c2 with delete_type TABLET_DATA_TOMBSTONED (TS 5dcc33cc621a4cbbae155f3230fb7c46 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35310
I20240503 02:00:16.549275 21066 tablet_replica.cc:330] T 8295ea038a88485095544cc608b000c2 P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 02:00:16.549476 21066 raft_consensus.cc:2229] T 8295ea038a88485095544cc608b000c2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:16.549679 21066 raft_consensus.cc:2258] T 8295ea038a88485095544cc608b000c2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:16.550412 21066 ts_tablet_manager.cc:1875] T 8295ea038a88485095544cc608b000c2 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:16.554819 21066 ts_tablet_manager.cc:1888] T 8295ea038a88485095544cc608b000c2 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:00:16.554981 21066 log.cc:1196] T 8295ea038a88485095544cc608b000c2 P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/8295ea038a88485095544cc608b000c2
I20240503 02:00:16.555794 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 8295ea038a88485095544cc608b000c2 (table rebalance_test_table_11 [id=dce2b86d058140cfb0a0d5088d32ec71]) successfully deleted
I20240503 02:00:16.697875 6861 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:16.713359 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:16.715204 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:16.726998 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:16.727444 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:16.743043 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:16.743711 9474 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:16.761162 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:16.842320 9362 consensus_queue.cc:237] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:16.844369 6883 raft_consensus.cc:1261] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:16.844924 9626 raft_consensus.cc:1261] T beb175f81d9e48229466503ebf7e3c09 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:16.845494 18823 consensus_queue.cc:1035] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:16.845882 19817 consensus_queue.cc:1035] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:16.849236 19817 raft_consensus.cc:2932] T beb175f81d9e48229466503ebf7e3c09 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:16.849965 6883 raft_consensus.cc:2932] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:16.853705 6736 catalog_manager.cc:5484] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:16.854701 9625 raft_consensus.cc:2932] T beb175f81d9e48229466503ebf7e3c09 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:16.858947 9474 tablet_service.cc:1514] Processing DeleteTablet for tablet beb175f81d9e48229466503ebf7e3c09 with delete_type TABLET_DATA_TOMBSTONED (TS bb5364d730cb4489a0bfc6c270b7450a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54678
I20240503 02:00:16.859380 21134 tablet_replica.cc:330] T beb175f81d9e48229466503ebf7e3c09 P bb5364d730cb4489a0bfc6c270b7450a: stopping tablet replica
I20240503 02:00:16.862457 21134 raft_consensus.cc:2229] T beb175f81d9e48229466503ebf7e3c09 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:16.862725 21134 raft_consensus.cc:2258] T beb175f81d9e48229466503ebf7e3c09 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:16.863425 21134 ts_tablet_manager.cc:1875] T beb175f81d9e48229466503ebf7e3c09 P bb5364d730cb4489a0bfc6c270b7450a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:16.868911 21134 ts_tablet_manager.cc:1888] T beb175f81d9e48229466503ebf7e3c09 P bb5364d730cb4489a0bfc6c270b7450a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:00:16.869105 21134 log.cc:1196] T beb175f81d9e48229466503ebf7e3c09 P bb5364d730cb4489a0bfc6c270b7450a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/beb175f81d9e48229466503ebf7e3c09
I20240503 02:00:16.869778 6725 catalog_manager.cc:4830] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): tablet beb175f81d9e48229466503ebf7e3c09 (table rebalancer_extra_table_20 [id=2d05875281f54da8b4f2477aaf91b494]) successfully deleted
I20240503 02:00:17.026453 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:17.032173 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:17.039392 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:17.043244 6861 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:17.056532 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:17.059214 9474 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:17.077174 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:17.090709 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:17.176884 9493 consensus_queue.cc:237] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:17.178483 9362 raft_consensus.cc:1261] T f20b6a14b8674e6fafefe20ca326d38b P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:17.178642 7143 raft_consensus.cc:1261] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:17.179010 21177 consensus_queue.cc:1035] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:17.179221 20969 consensus_queue.cc:1035] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:17.181311 21177 raft_consensus.cc:2932] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:17.181758 7145 raft_consensus.cc:2932] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:17.182754 9362 raft_consensus.cc:2932] T f20b6a14b8674e6fafefe20ca326d38b P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:17.185161 6737 catalog_manager.cc:5484] T f20b6a14b8674e6fafefe20ca326d38b P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: config changed from index 2 to 3, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:17.188180 9735 tablet_service.cc:1514] Processing DeleteTablet for tablet f20b6a14b8674e6fafefe20ca326d38b with delete_type TABLET_DATA_TOMBSTONED (TS db1a0000aaa346f79c0aa4b6c5105191 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:48152
I20240503 02:00:17.188735 21209 tablet_replica.cc:330] T f20b6a14b8674e6fafefe20ca326d38b P db1a0000aaa346f79c0aa4b6c5105191: stopping tablet replica
I20240503 02:00:17.189301 21209 raft_consensus.cc:2229] T f20b6a14b8674e6fafefe20ca326d38b P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:17.189523 21209 raft_consensus.cc:2258] T f20b6a14b8674e6fafefe20ca326d38b P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:17.190085 21209 ts_tablet_manager.cc:1875] T f20b6a14b8674e6fafefe20ca326d38b P db1a0000aaa346f79c0aa4b6c5105191: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:17.192684 21209 ts_tablet_manager.cc:1888] T f20b6a14b8674e6fafefe20ca326d38b P db1a0000aaa346f79c0aa4b6c5105191: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:00:17.192852 21209 log.cc:1196] T f20b6a14b8674e6fafefe20ca326d38b P db1a0000aaa346f79c0aa4b6c5105191: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/f20b6a14b8674e6fafefe20ca326d38b
I20240503 02:00:17.193532 6726 catalog_manager.cc:4830] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): tablet f20b6a14b8674e6fafefe20ca326d38b (table rebalancer_extra_table_34 [id=cb5e07c45e8b46888678a15d79f2cfd9]) successfully deleted
I20240503 02:00:17.351531 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:17.354003 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:17.368189 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:17.369405 9474 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:17.371249 6861 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:17.417609 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:17.436528 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:17.441917 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:17.503015 9626 consensus_queue.cc:237] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:17.504618 9493 raft_consensus.cc:1261] T 944657ceab07465092a950a8c793ceff P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer b7062fbbc2b54fff9e99ec4ea5562c76: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:17.504601 9755 raft_consensus.cc:1261] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer b7062fbbc2b54fff9e99ec4ea5562c76: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:17.505038 20025 consensus_queue.cc:1035] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:17.505242 21189 consensus_queue.cc:1035] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:17.507802 20025 raft_consensus.cc:2932] T 944657ceab07465092a950a8c793ceff P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:17.508159 9755 raft_consensus.cc:2932] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:17.513532 6740 catalog_manager.cc:5484] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: config changed from index 2 to 3, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New cstate: current_term: 1 leader_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:17.513684 9493 raft_consensus.cc:2932] T 944657ceab07465092a950a8c793ceff P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:17.517793 6861 tablet_service.cc:1514] Processing DeleteTablet for tablet 944657ceab07465092a950a8c793ceff with delete_type TABLET_DATA_TOMBSTONED (TS 5dcc33cc621a4cbbae155f3230fb7c46 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:35310
I20240503 02:00:17.524358 21280 tablet_replica.cc:330] T 944657ceab07465092a950a8c793ceff P 5dcc33cc621a4cbbae155f3230fb7c46: stopping tablet replica
I20240503 02:00:17.529206 21280 raft_consensus.cc:2229] T 944657ceab07465092a950a8c793ceff P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:17.529399 21280 raft_consensus.cc:2258] T 944657ceab07465092a950a8c793ceff P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:17.529850 21280 ts_tablet_manager.cc:1875] T 944657ceab07465092a950a8c793ceff P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:17.532927 21280 ts_tablet_manager.cc:1888] T 944657ceab07465092a950a8c793ceff P 5dcc33cc621a4cbbae155f3230fb7c46: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:00:17.533039 21280 log.cc:1196] T 944657ceab07465092a950a8c793ceff P 5dcc33cc621a4cbbae155f3230fb7c46: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/wals/944657ceab07465092a950a8c793ceff
I20240503 02:00:17.533844 6728 catalog_manager.cc:4830] TS 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): tablet 944657ceab07465092a950a8c793ceff (table rebalancer_extra_table_38 [id=2fc3f2f8c5d549c8b1f8c504355aede7]) successfully deleted
I20240503 02:00:17.686589 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:17.686738 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:17.692292 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:17.697569 6861 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:17.704207 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:17.712828 9474 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:17.714702 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:17.718971 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:17.818969 9755 consensus_queue.cc:237] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:17.820372 7012 raft_consensus.cc:1261] T 8feff37a8a6043d2b2f7581deb7c7934 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:17.820760 19514 consensus_queue.cc:1035] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:17.820901 9626 raft_consensus.cc:1261] T 8feff37a8a6043d2b2f7581deb7c7934 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:17.821322 19514 consensus_queue.cc:1035] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:17.823166 20843 raft_consensus.cc:2932] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:17.824009 9626 raft_consensus.cc:2932] T 8feff37a8a6043d2b2f7581deb7c7934 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:17.824949 7012 raft_consensus.cc:2932] T 8feff37a8a6043d2b2f7581deb7c7934 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:17.829234 6740 catalog_manager.cc:5484] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: config changed from index 2 to 3, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) evicted. New cstate: current_term: 1 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:17.833622 9341 tablet_service.cc:1514] Processing DeleteTablet for tablet 8feff37a8a6043d2b2f7581deb7c7934 with delete_type TABLET_DATA_TOMBSTONED (TS 12e8c683db054df3b75c6285b98a5ad8 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:36040
I20240503 02:00:17.834054 21357 tablet_replica.cc:330] T 8feff37a8a6043d2b2f7581deb7c7934 P 12e8c683db054df3b75c6285b98a5ad8: stopping tablet replica
I20240503 02:00:17.834414 21357 raft_consensus.cc:2229] T 8feff37a8a6043d2b2f7581deb7c7934 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:17.834618 21357 raft_consensus.cc:2258] T 8feff37a8a6043d2b2f7581deb7c7934 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:17.835167 21357 ts_tablet_manager.cc:1875] T 8feff37a8a6043d2b2f7581deb7c7934 P 12e8c683db054df3b75c6285b98a5ad8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:17.838470 21357 ts_tablet_manager.cc:1888] T 8feff37a8a6043d2b2f7581deb7c7934 P 12e8c683db054df3b75c6285b98a5ad8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:00:17.838652 21357 log.cc:1196] T 8feff37a8a6043d2b2f7581deb7c7934 P 12e8c683db054df3b75c6285b98a5ad8: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/wals/8feff37a8a6043d2b2f7581deb7c7934
I20240503 02:00:17.839542 6727 catalog_manager.cc:4830] TS 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): tablet 8feff37a8a6043d2b2f7581deb7c7934 (table rebalancer_extra_table_24 [id=5f00a869864944c1a935378f02b67261]) successfully deleted
I20240503 02:00:18.008293 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:18.008466 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:18.009274 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:18.010749 9474 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:18.017480 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:18.018723 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:18.039786 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:18.066012 6861 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:18.151307 7143 consensus_queue.cc:237] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:18.153448 7274 raft_consensus.cc:1261] T 75175ed25db846b9ba8e8e7b595102ca P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:18.154140 21061 consensus_queue.cc:1035] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:18.154630 9494 raft_consensus.cc:1261] T 75175ed25db846b9ba8e8e7b595102ca P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:18.155170 21292 consensus_queue.cc:1035] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:18.157101 21292 raft_consensus.cc:2932] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:18.158690 7274 raft_consensus.cc:2932] T 75175ed25db846b9ba8e8e7b595102ca P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:18.160310 9493 raft_consensus.cc:2932] T 75175ed25db846b9ba8e8e7b595102ca P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:18.161275 6736 catalog_manager.cc:5484] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index 2 to 3, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) evicted. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:18.166049 9735 tablet_service.cc:1514] Processing DeleteTablet for tablet 75175ed25db846b9ba8e8e7b595102ca with delete_type TABLET_DATA_TOMBSTONED (TS db1a0000aaa346f79c0aa4b6c5105191 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:48152
I20240503 02:00:18.166719 21430 tablet_replica.cc:330] T 75175ed25db846b9ba8e8e7b595102ca P db1a0000aaa346f79c0aa4b6c5105191: stopping tablet replica
I20240503 02:00:18.166915 21430 raft_consensus.cc:2229] T 75175ed25db846b9ba8e8e7b595102ca P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:18.167079 21430 raft_consensus.cc:2258] T 75175ed25db846b9ba8e8e7b595102ca P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:18.167801 21430 ts_tablet_manager.cc:1875] T 75175ed25db846b9ba8e8e7b595102ca P db1a0000aaa346f79c0aa4b6c5105191: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:18.170032 21430 ts_tablet_manager.cc:1888] T 75175ed25db846b9ba8e8e7b595102ca P db1a0000aaa346f79c0aa4b6c5105191: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:00:18.170115 21430 log.cc:1196] T 75175ed25db846b9ba8e8e7b595102ca P db1a0000aaa346f79c0aa4b6c5105191: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-7/wal/wals/75175ed25db846b9ba8e8e7b595102ca
I20240503 02:00:18.170663 6726 catalog_manager.cc:4830] TS db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): tablet 75175ed25db846b9ba8e8e7b595102ca (table rebalancer_extra_table_50 [id=6f17b2a98eb44079957e3e2ae659fb2c]) successfully deleted
I20240503 02:00:18.336432 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:18.337458 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:18.352600 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:18.362979 6861 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:18.381937 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:18.416008 9474 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:18.420924 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:18.446799 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:18.524276 9362 consensus_queue.cc:237] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:18.525632 9625 raft_consensus.cc:1261] T c4930b3ca84749f6a4cd245411f66075 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:18.525725 9756 raft_consensus.cc:1261] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:18.526013 19817 consensus_queue.cc:1035] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:18.526342 19817 consensus_queue.cc:1035] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:18.528090 21131 raft_consensus.cc:2932] T c4930b3ca84749f6a4cd245411f66075 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:18.528443 9625 raft_consensus.cc:2932] T c4930b3ca84749f6a4cd245411f66075 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:18.528632 9756 raft_consensus.cc:2932] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:18.532747 6736 catalog_manager.cc:5484] T c4930b3ca84749f6a4cd245411f66075 P b7062fbbc2b54fff9e99ec4ea5562c76 reported cstate change: config changed from index 2 to 3, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) evicted. New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:18.535856 7123 tablet_service.cc:1514] Processing DeleteTablet for tablet c4930b3ca84749f6a4cd245411f66075 with delete_type TABLET_DATA_TOMBSTONED (TS c688f3df56ac4b6d9b0c30fa4cc8ca2a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50370
I20240503 02:00:18.537537 21498 tablet_replica.cc:330] T c4930b3ca84749f6a4cd245411f66075 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: stopping tablet replica
I20240503 02:00:18.537796 21498 raft_consensus.cc:2229] T c4930b3ca84749f6a4cd245411f66075 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:18.537972 21498 raft_consensus.cc:2258] T c4930b3ca84749f6a4cd245411f66075 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:18.538599 21498 ts_tablet_manager.cc:1875] T c4930b3ca84749f6a4cd245411f66075 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:18.540913 21498 ts_tablet_manager.cc:1888] T c4930b3ca84749f6a4cd245411f66075 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:00:18.540997 21498 log.cc:1196] T c4930b3ca84749f6a4cd245411f66075 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/wals/c4930b3ca84749f6a4cd245411f66075
I20240503 02:00:18.542083 6727 catalog_manager.cc:4830] TS c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): tablet c4930b3ca84749f6a4cd245411f66075 (table rebalancer_extra_table_50 [id=6f17b2a98eb44079957e3e2ae659fb2c]) successfully deleted
I20240503 02:00:18.705441 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:18.707087 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:18.711071 6861 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:18.712392 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:18.717445 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:18.719561 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:18.726464 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:18.742192 9474 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:18.840276 6882 consensus_queue.cc:237] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:18.841958 9363 raft_consensus.cc:1261] T 3e824fcc9aac449d9373655b304010e2 P 12e8c683db054df3b75c6285b98a5ad8 [term 2 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:00:18.842043 7012 raft_consensus.cc:1261] T 3e824fcc9aac449d9373655b304010e2 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:00:18.842769 21176 consensus_queue.cc:1035] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:18.843031 20428 consensus_queue.cc:1035] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:18.844965 20428 raft_consensus.cc:2932] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:18.845618 7014 raft_consensus.cc:2932] T 3e824fcc9aac449d9373655b304010e2 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:18.845633 9363 raft_consensus.cc:2932] T 3e824fcc9aac449d9373655b304010e2 P 12e8c683db054df3b75c6285b98a5ad8 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:18.849870 6739 catalog_manager.cc:5484] T 3e824fcc9aac449d9373655b304010e2 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index 4 to 6, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New cstate: current_term: 2 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:18.853497 9474 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e824fcc9aac449d9373655b304010e2 with delete_type TABLET_DATA_TOMBSTONED (TS bb5364d730cb4489a0bfc6c270b7450a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54678
I20240503 02:00:18.853919 21577 tablet_replica.cc:330] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a: stopping tablet replica
I20240503 02:00:18.854157 21577 raft_consensus.cc:2229] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:18.854324 21577 raft_consensus.cc:2258] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:18.854849 21577 ts_tablet_manager.cc:1875] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:18.857372 21577 ts_tablet_manager.cc:1888] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 02:00:18.857549 21577 log.cc:1196] T 3e824fcc9aac449d9373655b304010e2 P bb5364d730cb4489a0bfc6c270b7450a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/3e824fcc9aac449d9373655b304010e2
I20240503 02:00:18.858208 6725 catalog_manager.cc:4830] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): tablet 3e824fcc9aac449d9373655b304010e2 (table rebalancer_extra_table_10 [id=15afeaf3f27142ebaf9ef93cac9e4dd6]) successfully deleted
I20240503 02:00:19.030241 9474 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:19.031301 6861 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:19.033005 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:19.038218 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:19.044158 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:19.054160 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:19.057821 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:19.077139 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:19.182888 7143 consensus_queue.cc:237] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:19.185106 9363 raft_consensus.cc:1261] T 38f126abb17c4654ae3cabe688d513d5 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:19.185626 21292 consensus_queue.cc:1035] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:19.185987 9755 raft_consensus.cc:1261] T 38f126abb17c4654ae3cabe688d513d5 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:19.186375 21292 consensus_queue.cc:1035] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:19.188014 21540 raft_consensus.cc:2932] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:19.190711 9756 raft_consensus.cc:2932] T 38f126abb17c4654ae3cabe688d513d5 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:19.195418 9363 raft_consensus.cc:2932] T 38f126abb17c4654ae3cabe688d513d5 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:19.196183 6733 catalog_manager.cc:5484] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index 2 to 3, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) evicted. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:19.202320 6992 tablet_service.cc:1514] Processing DeleteTablet for tablet 38f126abb17c4654ae3cabe688d513d5 with delete_type TABLET_DATA_TOMBSTONED (TS 63331f455be049c1bdd6c23c59ac1bef not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:35240
I20240503 02:00:19.203501 21655 tablet_replica.cc:330] T 38f126abb17c4654ae3cabe688d513d5 P 63331f455be049c1bdd6c23c59ac1bef: stopping tablet replica
I20240503 02:00:19.203683 21655 raft_consensus.cc:2229] T 38f126abb17c4654ae3cabe688d513d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:19.203827 21655 raft_consensus.cc:2258] T 38f126abb17c4654ae3cabe688d513d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:19.204355 21655 ts_tablet_manager.cc:1875] T 38f126abb17c4654ae3cabe688d513d5 P 63331f455be049c1bdd6c23c59ac1bef: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:19.207468 21655 ts_tablet_manager.cc:1888] T 38f126abb17c4654ae3cabe688d513d5 P 63331f455be049c1bdd6c23c59ac1bef: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:00:19.207551 21655 log.cc:1196] T 38f126abb17c4654ae3cabe688d513d5 P 63331f455be049c1bdd6c23c59ac1bef: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/wals/38f126abb17c4654ae3cabe688d513d5
I20240503 02:00:19.208185 6725 catalog_manager.cc:4830] TS 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): tablet 38f126abb17c4654ae3cabe688d513d5 (table rebalancer_extra_table_54 [id=8ade9b2b5b1a4a56a57cd8db33d169d2]) successfully deleted
I20240503 02:00:19.373188 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:19.389600 9474 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:19.395524 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:19.398911 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:19.399952 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:19.400305 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:19.413259 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:19.429081 6861 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:19.529819 7012 consensus_queue.cc:237] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:19.531618 6883 raft_consensus.cc:1261] T 1b7d65d849e242b4b6c0bc06422001db P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:19.531705 7274 raft_consensus.cc:1261] T 1b7d65d849e242b4b6c0bc06422001db P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:19.532184 21395 consensus_queue.cc:1035] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:19.532461 20276 consensus_queue.cc:1035] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:19.534845 21721 raft_consensus.cc:2932] T 1b7d65d849e242b4b6c0bc06422001db P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:19.535235 6883 raft_consensus.cc:2932] T 1b7d65d849e242b4b6c0bc06422001db P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:19.535235 7274 raft_consensus.cc:2932] T 1b7d65d849e242b4b6c0bc06422001db P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:19.539667 6739 catalog_manager.cc:5484] T 1b7d65d849e242b4b6c0bc06422001db P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index 2 to 3, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) evicted. New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:19.545449 9605 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b7d65d849e242b4b6c0bc06422001db with delete_type TABLET_DATA_TOMBSTONED (TS b7062fbbc2b54fff9e99ec4ea5562c76 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:58562
I20240503 02:00:19.546065 21727 tablet_replica.cc:330] T 1b7d65d849e242b4b6c0bc06422001db P b7062fbbc2b54fff9e99ec4ea5562c76: stopping tablet replica
I20240503 02:00:19.546309 21727 raft_consensus.cc:2229] T 1b7d65d849e242b4b6c0bc06422001db P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:19.546489 21727 raft_consensus.cc:2258] T 1b7d65d849e242b4b6c0bc06422001db P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:19.547266 21727 ts_tablet_manager.cc:1875] T 1b7d65d849e242b4b6c0bc06422001db P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:19.550077 21727 ts_tablet_manager.cc:1888] T 1b7d65d849e242b4b6c0bc06422001db P b7062fbbc2b54fff9e99ec4ea5562c76: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:00:19.550258 21727 log.cc:1196] T 1b7d65d849e242b4b6c0bc06422001db P b7062fbbc2b54fff9e99ec4ea5562c76: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/wals/1b7d65d849e242b4b6c0bc06422001db
I20240503 02:00:19.551064 6727 catalog_manager.cc:4830] TS b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): tablet 1b7d65d849e242b4b6c0bc06422001db (table rebalancer_extra_table_48 [id=b11be10400574bd0a4c4768ad34c1fc6]) successfully deleted
I20240503 02:00:19.690598 6861 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:19.696174 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:19.701705 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:19.709821 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:19.718760 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:19.719547 9474 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:19.722770 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:19.730463 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:19.822022 9363 consensus_queue.cc:237] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:19.823573 7143 raft_consensus.cc:1261] T b886572c6ec9473bbec744cee8de12f5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:19.823647 6883 raft_consensus.cc:1261] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:00:19.824074 21131 consensus_queue.cc:1035] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:19.824268 19817 consensus_queue.cc:1035] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:19.826408 21131 raft_consensus.cc:2932] T b886572c6ec9473bbec744cee8de12f5 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:19.826881 6883 raft_consensus.cc:2932] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:19.829483 7143 raft_consensus.cc:2932] T b886572c6ec9473bbec744cee8de12f5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:19.831485 6739 catalog_manager.cc:5484] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index 2 to 3, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) evicted. New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:19.838270 9474 tablet_service.cc:1514] Processing DeleteTablet for tablet b886572c6ec9473bbec744cee8de12f5 with delete_type TABLET_DATA_TOMBSTONED (TS bb5364d730cb4489a0bfc6c270b7450a not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54678
I20240503 02:00:19.838682 21800 tablet_replica.cc:330] T b886572c6ec9473bbec744cee8de12f5 P bb5364d730cb4489a0bfc6c270b7450a: stopping tablet replica
I20240503 02:00:19.838855 21800 raft_consensus.cc:2229] T b886572c6ec9473bbec744cee8de12f5 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:00:19.839043 21800 raft_consensus.cc:2258] T b886572c6ec9473bbec744cee8de12f5 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:00:19.839555 21800 ts_tablet_manager.cc:1875] T b886572c6ec9473bbec744cee8de12f5 P bb5364d730cb4489a0bfc6c270b7450a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:00:19.842291 21800 ts_tablet_manager.cc:1888] T b886572c6ec9473bbec744cee8de12f5 P bb5364d730cb4489a0bfc6c270b7450a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 02:00:19.842458 21800 log.cc:1196] T b886572c6ec9473bbec744cee8de12f5 P bb5364d730cb4489a0bfc6c270b7450a: Deleting WAL directory at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/wals/b886572c6ec9473bbec744cee8de12f5
I20240503 02:00:19.843114 6725 catalog_manager.cc:4830] TS bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): tablet b886572c6ec9473bbec744cee8de12f5 (table rebalancer_extra_table_30 [id=70efe522f8fa4d9bb92406e8bfcb0ef3]) successfully deleted
I20240503 02:00:20.009032 7012 consensus_queue.cc:237] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:20.011298 9626 raft_consensus.cc:1261] T b5329f65c13c47d485cee81cf9cf4f1e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:20.011777 6950 consensus_peers.cc:485] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef -> Peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): Couldn't send request to peer 5dcc33cc621a4cbbae155f3230fb7c46. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5329f65c13c47d485cee81cf9cf4f1e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:20.011755 9756 raft_consensus.cc:1261] T b5329f65c13c47d485cee81cf9cf4f1e P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:20.012029 21395 consensus_queue.cc:1035] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:20.012571 21395 consensus_queue.cc:1035] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:20.014479 21721 raft_consensus.cc:2932] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:20.015108 9755 raft_consensus.cc:2932] T b5329f65c13c47d485cee81cf9cf4f1e P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:20.016824 9625 raft_consensus.cc:2932] T b5329f65c13c47d485cee81cf9cf4f1e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:20.018842 6732 catalog_manager.cc:5484] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:20.056701 7012 consensus_queue.cc:237] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
W20240503 02:00:20.058614 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 85d3e89fac21402387aa812f754cf5bc. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:20.058938 7143 raft_consensus.cc:1261] T 85d3e89fac21402387aa812f754cf5bc P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:20.058872 9362 raft_consensus.cc:1261] T 85d3e89fac21402387aa812f754cf5bc P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:20.059386 21395 consensus_queue.cc:1035] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:20.059810 21395 consensus_queue.cc:1035] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:20.063145 20276 raft_consensus.cc:2932] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:20.063700 7143 raft_consensus.cc:2932] T 85d3e89fac21402387aa812f754cf5bc P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:20.066800 9362 raft_consensus.cc:2932] T 85d3e89fac21402387aa812f754cf5bc P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:20.068821 6732 catalog_manager.cc:5484] T 85d3e89fac21402387aa812f754cf5bc P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index 5 to 6, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:20.121274 7012 consensus_queue.cc:237] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:20.123098 9362 raft_consensus.cc:1261] T 822cbc676b234eda8770f7b485ebe302 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:20.123159 9626 raft_consensus.cc:1261] T 822cbc676b234eda8770f7b485ebe302 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:20.123642 21395 consensus_queue.cc:1035] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:20.125665 21395 consensus_queue.cc:1035] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:00:20.126062 6948 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): Couldn't send request to peer 1d0e738c34af40fa9950c6f086619b3a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 822cbc676b234eda8770f7b485ebe302. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:20.131754 21395 raft_consensus.cc:2932] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:20.135706 6736 catalog_manager.cc:5484] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:20.138748 9362 raft_consensus.cc:2932] T 822cbc676b234eda8770f7b485ebe302 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:20.143430 9626 raft_consensus.cc:2932] T 822cbc676b234eda8770f7b485ebe302 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:20.168103 6883 consensus_queue.cc:237] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:20.169791 9362 raft_consensus.cc:1261] T c4ac4ad7f9eb4723814f65ce96758c0e P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 02:00:20.169987 6816 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c4ac4ad7f9eb4723814f65ce96758c0e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:20.170089 7014 raft_consensus.cc:1261] T c4ac4ad7f9eb4723814f65ce96758c0e P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:20.170469 21176 consensus_queue.cc:1035] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:20.170783 20428 consensus_queue.cc:1035] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:20.173790 21176 raft_consensus.cc:2932] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 02:00:20.174661 7014 raft_consensus.cc:2932] T c4ac4ad7f9eb4723814f65ce96758c0e P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 02:00:20.177681 9362 raft_consensus.cc:2932] T c4ac4ad7f9eb4723814f65ce96758c0e P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 02:00:20.178687 6736 catalog_manager.cc:5484] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index 5 to 6, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 02:00:20.212177 9494 consensus_queue.cc:237] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:20.213780 9755 raft_consensus.cc:1261] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:20.214023 9430 consensus_peers.cc:485] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a -> Peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): Couldn't send request to peer 5dcc33cc621a4cbbae155f3230fb7c46. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6eeb7c5fa08c48c08c36bf03bd1c85ae. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:20.214262 21966 consensus_queue.cc:1035] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:20.214388 7274 raft_consensus.cc:1261] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:20.214870 21965 consensus_queue.cc:1035] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:20.218542 7274 raft_consensus.cc:2932] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:20.217955 21965 raft_consensus.cc:2932] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:20.222364 9755 raft_consensus.cc:2932] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:20.222344 6732 catalog_manager.cc:5484] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:20.252800 6883 consensus_queue.cc:237] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:20.254909 9362 raft_consensus.cc:1261] T 72e090a6c18e4a13bbb343eb40a07694 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:20.254886 7143 raft_consensus.cc:1261] T 72e090a6c18e4a13bbb343eb40a07694 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 02:00:20.255434 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72e090a6c18e4a13bbb343eb40a07694. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:20.255998 21775 consensus_queue.cc:1035] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:20.256309 21176 consensus_queue.cc:1035] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:20.258476 20428 raft_consensus.cc:2932] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:20.258929 9362 raft_consensus.cc:2932] T 72e090a6c18e4a13bbb343eb40a07694 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:20.262671 7143 raft_consensus.cc:2932] T 72e090a6c18e4a13bbb343eb40a07694 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:20.263870 6736 catalog_manager.cc:5484] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index 5 to 6, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:20.295887 6883 consensus_queue.cc:237] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:20.297493 9362 raft_consensus.cc:1261] T 9b9405669fb640c4a3a8fe2f31af291e P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:20.297624 9626 raft_consensus.cc:1261] T 9b9405669fb640c4a3a8fe2f31af291e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:20.298019 6817 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271): Couldn't send request to peer 1d0e738c34af40fa9950c6f086619b3a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b9405669fb640c4a3a8fe2f31af291e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:20.298054 20428 consensus_queue.cc:1035] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:20.298368 21176 consensus_queue.cc:1035] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:20.300738 21176 raft_consensus.cc:2932] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:20.301244 9626 raft_consensus.cc:2932] T 9b9405669fb640c4a3a8fe2f31af291e P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:20.303491 9362 raft_consensus.cc:2932] T 9b9405669fb640c4a3a8fe2f31af291e P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:20.307687 6732 catalog_manager.cc:5484] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: config changed from index -1 to 2, VOTER 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132) added. New cstate: current_term: 1 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:20.339037 9362 consensus_queue.cc:237] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:20.341223 9756 raft_consensus.cc:1261] T 02e8cbe02d1b4017909e9bfddaf093d8 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:20.341223 7272 raft_consensus.cc:1261] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer 12e8c683db054df3b75c6285b98a5ad8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:20.341359 9296 consensus_peers.cc:485] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8 -> Peer 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231): Couldn't send request to peer 63331f455be049c1bdd6c23c59ac1bef. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02e8cbe02d1b4017909e9bfddaf093d8. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:20.341660 22022 consensus_queue.cc:1035] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:20.341953 21794 consensus_queue.cc:1035] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:20.349380 21794 raft_consensus.cc:2932] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:20.352131 9756 raft_consensus.cc:2932] T 02e8cbe02d1b4017909e9bfddaf093d8 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:20.353268 6736 catalog_manager.cc:5484] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New cstate: current_term: 1 leader_uuid: "12e8c683db054df3b75c6285b98a5ad8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:20.354982 7272 raft_consensus.cc:2932] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:20.388301 7272 consensus_queue.cc:237] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:20.390285 6882 raft_consensus.cc:1261] T 4600418bf64c4a0d962831c6924c46e1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 02:00:20.390334 7209 consensus_peers.cc:485] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4600418bf64c4a0d962831c6924c46e1. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:20.390766 9756 raft_consensus.cc:1261] T 4600418bf64c4a0d962831c6924c46e1 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:20.390906 21630 consensus_queue.cc:1035] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:20.391211 20988 consensus_queue.cc:1035] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:20.393815 20988 raft_consensus.cc:2932] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 02:00:20.394707 9756 raft_consensus.cc:2932] T 4600418bf64c4a0d962831c6924c46e1 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 02:00:20.397567 6882 raft_consensus.cc:2932] T 4600418bf64c4a0d962831c6924c46e1 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 02:00:20.398401 6736 catalog_manager.cc:5484] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: config changed from index 5 to 6, VOTER bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) added. New cstate: current_term: 1 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } }
I20240503 02:00:20.430434 9494 consensus_queue.cc:237] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:20.432276 9362 raft_consensus.cc:1261] T df13afe0f90c4c24b66026048b5852ec P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:20.432271 9626 raft_consensus.cc:1261] T df13afe0f90c4c24b66026048b5852ec P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:20.432580 9429 consensus_peers.cc:485] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a -> Peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): Couldn't send request to peer c688f3df56ac4b6d9b0c30fa4cc8ca2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df13afe0f90c4c24b66026048b5852ec. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:20.432796 21965 consensus_queue.cc:1035] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:20.433111 21641 consensus_queue.cc:1035] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:20.436869 21965 raft_consensus.cc:2932] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:20.437228 9626 raft_consensus.cc:2932] T df13afe0f90c4c24b66026048b5852ec P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:20.439350 9362 raft_consensus.cc:2932] T df13afe0f90c4c24b66026048b5852ec P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:20.442132 6737 catalog_manager.cc:5484] T df13afe0f90c4c24b66026048b5852ec P b7062fbbc2b54fff9e99ec4ea5562c76 reported cstate change: config changed from index -1 to 2, VOTER c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131) added. New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:20.471824 7143 consensus_queue.cc:237] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:20.474318 7272 raft_consensus.cc:1261] T 822bafcbc6de406997955ff7a1d9b612 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240503 02:00:20.474551 7080 consensus_peers.cc:485] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 822bafcbc6de406997955ff7a1d9b612. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:20.474591 9494 raft_consensus.cc:1261] T 822bafcbc6de406997955ff7a1d9b612 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:00:20.474818 21540 consensus_queue.cc:1035] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:20.475157 21292 consensus_queue.cc:1035] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:00:20.477969 21292 raft_consensus.cc:2932] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:20.478698 9494 raft_consensus.cc:2932] T 822bafcbc6de406997955ff7a1d9b612 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:20.481560 7272 raft_consensus.cc:2932] T 822bafcbc6de406997955ff7a1d9b612 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:20.481982 6739 catalog_manager.cc:5484] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index 5 to 6, VOTER 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) added. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:20.514377 9494 consensus_queue.cc:237] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:20.516608 9626 raft_consensus.cc:1261] T 49eabbb8972840759ffa83951cb0b3b7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:20.516790 9430 consensus_peers.cc:485] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a -> Peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): Couldn't send request to peer 5dcc33cc621a4cbbae155f3230fb7c46. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49eabbb8972840759ffa83951cb0b3b7. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:20.516894 9756 raft_consensus.cc:1261] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Refusing update from remote peer bb5364d730cb4489a0bfc6c270b7450a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:20.517022 21641 consensus_queue.cc:1035] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:20.517499 21641 consensus_queue.cc:1035] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:20.519869 21965 raft_consensus.cc:2932] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:20.520309 9756 raft_consensus.cc:2932] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:20.522748 9626 raft_consensus.cc:2932] T 49eabbb8972840759ffa83951cb0b3b7 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:20.524011 6732 catalog_manager.cc:5484] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a reported cstate change: config changed from index -1 to 2, VOTER 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129) added. New cstate: current_term: 1 leader_uuid: "bb5364d730cb4489a0bfc6c270b7450a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:20.557124 7143 consensus_queue.cc:237] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:20.558874 7012 raft_consensus.cc:1261] T c6833c997d284063955d0b9b09ccb3b7 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
W20240503 02:00:20.558933 7080 consensus_peers.cc:485] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6833c997d284063955d0b9b09ccb3b7. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:20.559271 9756 raft_consensus.cc:1261] T c6833c997d284063955d0b9b09ccb3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 2 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:00:20.559556 21292 consensus_queue.cc:1035] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:20.559724 22081 consensus_queue.cc:1035] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:00:20.564019 22081 raft_consensus.cc:2932] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:20.564456 7012 raft_consensus.cc:2932] T c6833c997d284063955d0b9b09ccb3b7 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:20.567032 9756 raft_consensus.cc:2932] T c6833c997d284063955d0b9b09ccb3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:20.568506 6739 catalog_manager.cc:5484] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: config changed from index 6 to 7, VOTER b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) added. New cstate: current_term: 2 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:20.575752 22126 ts_tablet_manager.cc:902] T 85d3e89fac21402387aa812f754cf5bc P b7062fbbc2b54fff9e99ec4ea5562c76: Initiating tablet copy from peer 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 02:00:20.576332 22126 tablet_copy_client.cc:308] T 85d3e89fac21402387aa812f754cf5bc P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:37231
I20240503 02:00:20.576833 7034 tablet_copy_service.cc:140] P 63331f455be049c1bdd6c23c59ac1bef: Received BeginTabletCopySession request for tablet 85d3e89fac21402387aa812f754cf5bc from peer b7062fbbc2b54fff9e99ec4ea5562c76 ({username='slave'} at 127.0.99.135:36345)
I20240503 02:00:20.576932 7034 tablet_copy_service.cc:161] P 63331f455be049c1bdd6c23c59ac1bef: Beginning new tablet copy session on tablet 85d3e89fac21402387aa812f754cf5bc from peer b7062fbbc2b54fff9e99ec4ea5562c76 at {username='slave'} at 127.0.99.135:36345: session id = b7062fbbc2b54fff9e99ec4ea5562c76-85d3e89fac21402387aa812f754cf5bc
I20240503 02:00:20.577762 7034 tablet_copy_source_session.cc:205] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:20.578660 22126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85d3e89fac21402387aa812f754cf5bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.582000 22126 tablet_copy_client.cc:783] T 85d3e89fac21402387aa812f754cf5bc P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:20.582295 22126 tablet_copy_client.cc:647] T 85d3e89fac21402387aa812f754cf5bc P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:20.585034 22126 tablet_copy_client.cc:523] T 85d3e89fac21402387aa812f754cf5bc P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:20.586831 22126 tablet_bootstrap.cc:492] T 85d3e89fac21402387aa812f754cf5bc P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 02:00:20.609184 22142 ts_tablet_manager.cc:902] T b5329f65c13c47d485cee81cf9cf4f1e P 5dcc33cc621a4cbbae155f3230fb7c46: Initiating tablet copy from peer 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 02:00:20.609773 22142 tablet_copy_client.cc:308] T b5329f65c13c47d485cee81cf9cf4f1e P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:37231
I20240503 02:00:20.610553 7034 tablet_copy_service.cc:140] P 63331f455be049c1bdd6c23c59ac1bef: Received BeginTabletCopySession request for tablet b5329f65c13c47d485cee81cf9cf4f1e from peer 5dcc33cc621a4cbbae155f3230fb7c46 ({username='slave'} at 127.0.99.129:33603)
I20240503 02:00:20.610666 7034 tablet_copy_service.cc:161] P 63331f455be049c1bdd6c23c59ac1bef: Beginning new tablet copy session on tablet b5329f65c13c47d485cee81cf9cf4f1e from peer 5dcc33cc621a4cbbae155f3230fb7c46 at {username='slave'} at 127.0.99.129:33603: session id = 5dcc33cc621a4cbbae155f3230fb7c46-b5329f65c13c47d485cee81cf9cf4f1e
I20240503 02:00:20.611197 7143 consensus_queue.cc:237] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:20.611655 7034 tablet_copy_source_session.cc:205] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:20.612671 9494 raft_consensus.cc:1261] T c43f13a33fe94033bce9debbae6990c2 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:00:20.612895 7079 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053): Couldn't send request to peer db1a0000aaa346f79c0aa4b6c5105191. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c43f13a33fe94033bce9debbae6990c2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:20.612968 9363 raft_consensus.cc:1261] T c43f13a33fe94033bce9debbae6990c2 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:00:20.613112 22120 consensus_queue.cc:1035] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:20.613353 21292 consensus_queue.cc:1035] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:20.613461 22142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5329f65c13c47d485cee81cf9cf4f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.616071 22120 raft_consensus.cc:2932] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:20.616469 22142 tablet_copy_client.cc:783] T b5329f65c13c47d485cee81cf9cf4f1e P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:20.616680 22142 tablet_copy_client.cc:647] T b5329f65c13c47d485cee81cf9cf4f1e P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:20.617892 22142 tablet_copy_client.cc:523] T b5329f65c13c47d485cee81cf9cf4f1e P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:20.618129 9494 raft_consensus.cc:2932] T c43f13a33fe94033bce9debbae6990c2 P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:20.618844 9363 raft_consensus.cc:2932] T c43f13a33fe94033bce9debbae6990c2 P 12e8c683db054df3b75c6285b98a5ad8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:20.620029 22142 tablet_bootstrap.cc:492] T b5329f65c13c47d485cee81cf9cf4f1e P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 02:00:20.621748 22126 tablet_bootstrap.cc:492] T 85d3e89fac21402387aa812f754cf5bc P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:20.622068 22126 tablet_bootstrap.cc:492] T 85d3e89fac21402387aa812f754cf5bc P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap complete.
I20240503 02:00:20.622267 22126 ts_tablet_manager.cc:1367] T 85d3e89fac21402387aa812f754cf5bc P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.035s user 0.029s sys 0.000s
I20240503 02:00:20.622761 22126 raft_consensus.cc:348] T 85d3e89fac21402387aa812f754cf5bc P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:20.622895 22126 raft_consensus.cc:729] T 85d3e89fac21402387aa812f754cf5bc P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.623057 22126 consensus_queue.cc:260] T 85d3e89fac21402387aa812f754cf5bc P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:20.623476 22126 ts_tablet_manager.cc:1398] T 85d3e89fac21402387aa812f754cf5bc P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 02:00:20.624408 6732 catalog_manager.cc:5484] T c43f13a33fe94033bce9debbae6990c2 P 12e8c683db054df3b75c6285b98a5ad8 reported cstate change: config changed from index -1 to 2, VOTER db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136) added. New cstate: current_term: 1 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:20.625257 7034 tablet_copy_service.cc:342] P 63331f455be049c1bdd6c23c59ac1bef: Request end of tablet copy session b7062fbbc2b54fff9e99ec4ea5562c76-85d3e89fac21402387aa812f754cf5bc received from {username='slave'} at 127.0.99.135:36345
I20240503 02:00:20.625344 7034 tablet_copy_service.cc:433] P 63331f455be049c1bdd6c23c59ac1bef: ending tablet copy session b7062fbbc2b54fff9e99ec4ea5562c76-85d3e89fac21402387aa812f754cf5bc on tablet 85d3e89fac21402387aa812f754cf5bc with peer b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:20.637277 22142 tablet_bootstrap.cc:492] T b5329f65c13c47d485cee81cf9cf4f1e P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:20.637835 22142 tablet_bootstrap.cc:492] T b5329f65c13c47d485cee81cf9cf4f1e P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap complete.
I20240503 02:00:20.638089 22142 ts_tablet_manager.cc:1367] T b5329f65c13c47d485cee81cf9cf4f1e P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240503 02:00:20.638639 22142 raft_consensus.cc:348] T b5329f65c13c47d485cee81cf9cf4f1e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:20.638754 22142 raft_consensus.cc:729] T b5329f65c13c47d485cee81cf9cf4f1e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.639050 22142 consensus_queue.cc:260] T b5329f65c13c47d485cee81cf9cf4f1e P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:20.639364 22142 ts_tablet_manager.cc:1398] T b5329f65c13c47d485cee81cf9cf4f1e P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.001s user 0.000s sys 0.003s
I20240503 02:00:20.640075 7034 tablet_copy_service.cc:342] P 63331f455be049c1bdd6c23c59ac1bef: Request end of tablet copy session 5dcc33cc621a4cbbae155f3230fb7c46-b5329f65c13c47d485cee81cf9cf4f1e received from {username='slave'} at 127.0.99.129:33603
I20240503 02:00:20.640197 7034 tablet_copy_service.cc:433] P 63331f455be049c1bdd6c23c59ac1bef: ending tablet copy session 5dcc33cc621a4cbbae155f3230fb7c46-b5329f65c13c47d485cee81cf9cf4f1e on tablet b5329f65c13c47d485cee81cf9cf4f1e with peer 5dcc33cc621a4cbbae155f3230fb7c46
I20240503 02:00:20.690711 22192 ts_tablet_manager.cc:902] T 822cbc676b234eda8770f7b485ebe302 P 1d0e738c34af40fa9950c6f086619b3a: Initiating tablet copy from peer 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 02:00:20.692812 22192 tablet_copy_client.cc:308] T 822cbc676b234eda8770f7b485ebe302 P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:37231
I20240503 02:00:20.693945 7034 tablet_copy_service.cc:140] P 63331f455be049c1bdd6c23c59ac1bef: Received BeginTabletCopySession request for tablet 822cbc676b234eda8770f7b485ebe302 from peer 1d0e738c34af40fa9950c6f086619b3a ({username='slave'} at 127.0.99.132:48463)
I20240503 02:00:20.694187 7034 tablet_copy_service.cc:161] P 63331f455be049c1bdd6c23c59ac1bef: Beginning new tablet copy session on tablet 822cbc676b234eda8770f7b485ebe302 from peer 1d0e738c34af40fa9950c6f086619b3a at {username='slave'} at 127.0.99.132:48463: session id = 1d0e738c34af40fa9950c6f086619b3a-822cbc676b234eda8770f7b485ebe302
I20240503 02:00:20.695612 7034 tablet_copy_source_session.cc:205] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:20.696681 22192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 822cbc676b234eda8770f7b485ebe302. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.702306 22192 tablet_copy_client.cc:783] T 822cbc676b234eda8770f7b485ebe302 P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:20.702564 22192 tablet_copy_client.cc:647] T 822cbc676b234eda8770f7b485ebe302 P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:20.703785 22192 tablet_copy_client.cc:523] T 822cbc676b234eda8770f7b485ebe302 P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:20.705391 22192 tablet_bootstrap.cc:492] T 822cbc676b234eda8770f7b485ebe302 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
I20240503 02:00:20.710963 22192 tablet_bootstrap.cc:492] T 822cbc676b234eda8770f7b485ebe302 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:20.711319 22192 tablet_bootstrap.cc:492] T 822cbc676b234eda8770f7b485ebe302 P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap complete.
I20240503 02:00:20.711527 22192 ts_tablet_manager.cc:1367] T 822cbc676b234eda8770f7b485ebe302 P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240503 02:00:20.712072 22192 raft_consensus.cc:348] T 822cbc676b234eda8770f7b485ebe302 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:20.712213 22192 raft_consensus.cc:729] T 822cbc676b234eda8770f7b485ebe302 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.712359 22192 consensus_queue.cc:260] T 822cbc676b234eda8770f7b485ebe302 P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:20.712796 22192 ts_tablet_manager.cc:1398] T 822cbc676b234eda8770f7b485ebe302 P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:20.713330 7034 tablet_copy_service.cc:342] P 63331f455be049c1bdd6c23c59ac1bef: Request end of tablet copy session 1d0e738c34af40fa9950c6f086619b3a-822cbc676b234eda8770f7b485ebe302 received from {username='slave'} at 127.0.99.132:48463
I20240503 02:00:20.713426 7034 tablet_copy_service.cc:433] P 63331f455be049c1bdd6c23c59ac1bef: ending tablet copy session 1d0e738c34af40fa9950c6f086619b3a-822cbc676b234eda8770f7b485ebe302 on tablet 822cbc676b234eda8770f7b485ebe302 with peer 1d0e738c34af40fa9950c6f086619b3a
I20240503 02:00:20.730508 22198 ts_tablet_manager.cc:902] T c4ac4ad7f9eb4723814f65ce96758c0e P bb5364d730cb4489a0bfc6c270b7450a: Initiating tablet copy from peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 02:00:20.731763 22198 tablet_copy_client.cc:308] T c4ac4ad7f9eb4723814f65ce96758c0e P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:42943
I20240503 02:00:20.732254 6903 tablet_copy_service.cc:140] P 5dcc33cc621a4cbbae155f3230fb7c46: Received BeginTabletCopySession request for tablet c4ac4ad7f9eb4723814f65ce96758c0e from peer bb5364d730cb4489a0bfc6c270b7450a ({username='slave'} at 127.0.99.134:59615)
I20240503 02:00:20.732376 6903 tablet_copy_service.cc:161] P 5dcc33cc621a4cbbae155f3230fb7c46: Beginning new tablet copy session on tablet c4ac4ad7f9eb4723814f65ce96758c0e from peer bb5364d730cb4489a0bfc6c270b7450a at {username='slave'} at 127.0.99.134:59615: session id = bb5364d730cb4489a0bfc6c270b7450a-c4ac4ad7f9eb4723814f65ce96758c0e
I20240503 02:00:20.733464 6903 tablet_copy_source_session.cc:205] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:20.735328 22198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ac4ad7f9eb4723814f65ce96758c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.740276 22198 tablet_copy_client.cc:783] T c4ac4ad7f9eb4723814f65ce96758c0e P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:20.741537 22198 tablet_copy_client.cc:647] T c4ac4ad7f9eb4723814f65ce96758c0e P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:20.744282 22198 tablet_copy_client.cc:523] T c4ac4ad7f9eb4723814f65ce96758c0e P bb5364d730cb4489a0bfc6c270b7450a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:20.746382 22198 tablet_bootstrap.cc:492] T c4ac4ad7f9eb4723814f65ce96758c0e P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap starting.
I20240503 02:00:20.788821 22198 tablet_bootstrap.cc:492] T c4ac4ad7f9eb4723814f65ce96758c0e P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:20.789201 22198 tablet_bootstrap.cc:492] T c4ac4ad7f9eb4723814f65ce96758c0e P bb5364d730cb4489a0bfc6c270b7450a: Bootstrap complete.
I20240503 02:00:20.789398 22198 ts_tablet_manager.cc:1367] T c4ac4ad7f9eb4723814f65ce96758c0e P bb5364d730cb4489a0bfc6c270b7450a: Time spent bootstrapping tablet: real 0.043s user 0.030s sys 0.001s
I20240503 02:00:20.789902 22198 raft_consensus.cc:348] T c4ac4ad7f9eb4723814f65ce96758c0e P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:20.790016 22198 raft_consensus.cc:729] T c4ac4ad7f9eb4723814f65ce96758c0e P bb5364d730cb4489a0bfc6c270b7450a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5364d730cb4489a0bfc6c270b7450a, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.790160 22198 consensus_queue.cc:260] T c4ac4ad7f9eb4723814f65ce96758c0e P bb5364d730cb4489a0bfc6c270b7450a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:20.790568 22198 ts_tablet_manager.cc:1398] T c4ac4ad7f9eb4723814f65ce96758c0e P bb5364d730cb4489a0bfc6c270b7450a: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 02:00:20.791268 6903 tablet_copy_service.cc:342] P 5dcc33cc621a4cbbae155f3230fb7c46: Request end of tablet copy session bb5364d730cb4489a0bfc6c270b7450a-c4ac4ad7f9eb4723814f65ce96758c0e received from {username='slave'} at 127.0.99.134:59615
I20240503 02:00:20.791364 6903 tablet_copy_service.cc:433] P 5dcc33cc621a4cbbae155f3230fb7c46: ending tablet copy session bb5364d730cb4489a0bfc6c270b7450a-c4ac4ad7f9eb4723814f65ce96758c0e on tablet c4ac4ad7f9eb4723814f65ce96758c0e with peer bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:20.814877 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:20.817270 22126 ts_tablet_manager.cc:902] T 72e090a6c18e4a13bbb343eb40a07694 P b7062fbbc2b54fff9e99ec4ea5562c76: Initiating tablet copy from peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 02:00:20.817833 22126 tablet_copy_client.cc:308] T 72e090a6c18e4a13bbb343eb40a07694 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:42943
I20240503 02:00:20.818284 6903 tablet_copy_service.cc:140] P 5dcc33cc621a4cbbae155f3230fb7c46: Received BeginTabletCopySession request for tablet 72e090a6c18e4a13bbb343eb40a07694 from peer b7062fbbc2b54fff9e99ec4ea5562c76 ({username='slave'} at 127.0.99.135:54599)
I20240503 02:00:20.818399 6903 tablet_copy_service.cc:161] P 5dcc33cc621a4cbbae155f3230fb7c46: Beginning new tablet copy session on tablet 72e090a6c18e4a13bbb343eb40a07694 from peer b7062fbbc2b54fff9e99ec4ea5562c76 at {username='slave'} at 127.0.99.135:54599: session id = b7062fbbc2b54fff9e99ec4ea5562c76-72e090a6c18e4a13bbb343eb40a07694
I20240503 02:00:20.819538 6903 tablet_copy_source_session.cc:205] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:20.826467 22142 ts_tablet_manager.cc:902] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 5dcc33cc621a4cbbae155f3230fb7c46: Initiating tablet copy from peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 02:00:20.827018 22142 tablet_copy_client.cc:308] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:43471
I20240503 02:00:20.827639 22126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e090a6c18e4a13bbb343eb40a07694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.827630 9514 tablet_copy_service.cc:140] P bb5364d730cb4489a0bfc6c270b7450a: Received BeginTabletCopySession request for tablet 6eeb7c5fa08c48c08c36bf03bd1c85ae from peer 5dcc33cc621a4cbbae155f3230fb7c46 ({username='slave'} at 127.0.99.129:38243)
I20240503 02:00:20.827726 9514 tablet_copy_service.cc:161] P bb5364d730cb4489a0bfc6c270b7450a: Beginning new tablet copy session on tablet 6eeb7c5fa08c48c08c36bf03bd1c85ae from peer 5dcc33cc621a4cbbae155f3230fb7c46 at {username='slave'} at 127.0.99.129:38243: session id = 5dcc33cc621a4cbbae155f3230fb7c46-6eeb7c5fa08c48c08c36bf03bd1c85ae
I20240503 02:00:20.828729 9514 tablet_copy_source_session.cc:205] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P bb5364d730cb4489a0bfc6c270b7450a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:20.830396 22126 tablet_copy_client.cc:783] T 72e090a6c18e4a13bbb343eb40a07694 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:20.830430 22142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eeb7c5fa08c48c08c36bf03bd1c85ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.830583 22126 tablet_copy_client.cc:647] T 72e090a6c18e4a13bbb343eb40a07694 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:20.832330 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:20.838455 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:20.841641 9474 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:00:20.848206 9605 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:20.849864 22126 tablet_copy_client.cc:523] T 72e090a6c18e4a13bbb343eb40a07694 P b7062fbbc2b54fff9e99ec4ea5562c76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:20.851609 22126 tablet_bootstrap.cc:492] T 72e090a6c18e4a13bbb343eb40a07694 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap starting.
I20240503 02:00:20.859086 22226 ts_tablet_manager.cc:902] T 02e8cbe02d1b4017909e9bfddaf093d8 P 63331f455be049c1bdd6c23c59ac1bef: Initiating tablet copy from peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:20.859764 22226 tablet_copy_client.cc:308] T 02e8cbe02d1b4017909e9bfddaf093d8 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.133:37387
I20240503 02:00:20.862654 9383 tablet_copy_service.cc:140] P 12e8c683db054df3b75c6285b98a5ad8: Received BeginTabletCopySession request for tablet 02e8cbe02d1b4017909e9bfddaf093d8 from peer 63331f455be049c1bdd6c23c59ac1bef ({username='slave'} at 127.0.99.130:52605)
I20240503 02:00:20.862787 9383 tablet_copy_service.cc:161] P 12e8c683db054df3b75c6285b98a5ad8: Beginning new tablet copy session on tablet 02e8cbe02d1b4017909e9bfddaf093d8 from peer 63331f455be049c1bdd6c23c59ac1bef at {username='slave'} at 127.0.99.130:52605: session id = 63331f455be049c1bdd6c23c59ac1bef-02e8cbe02d1b4017909e9bfddaf093d8
I20240503 02:00:20.863965 9383 tablet_copy_source_session.cc:205] T 02e8cbe02d1b4017909e9bfddaf093d8 P 12e8c683db054df3b75c6285b98a5ad8: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:20.866353 6861 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:20.866742 22142 tablet_copy_client.cc:783] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:20.866911 22142 tablet_copy_client.cc:647] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:20.870904 22142 tablet_copy_client.cc:523] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:20.871934 9341 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:20.874977 22226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02e8cbe02d1b4017909e9bfddaf093d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.877578 22226 tablet_copy_client.cc:783] T 02e8cbe02d1b4017909e9bfddaf093d8 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:20.877789 22226 tablet_copy_client.cc:647] T 02e8cbe02d1b4017909e9bfddaf093d8 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:20.882436 22142 tablet_bootstrap.cc:492] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
I20240503 02:00:20.882764 22226 tablet_copy_client.cc:523] T 02e8cbe02d1b4017909e9bfddaf093d8 P 63331f455be049c1bdd6c23c59ac1bef: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:20.884558 22226 tablet_bootstrap.cc:492] T 02e8cbe02d1b4017909e9bfddaf093d8 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap starting.
I20240503 02:00:20.893671 22226 tablet_bootstrap.cc:492] T 02e8cbe02d1b4017909e9bfddaf093d8 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:20.894140 22226 tablet_bootstrap.cc:492] T 02e8cbe02d1b4017909e9bfddaf093d8 P 63331f455be049c1bdd6c23c59ac1bef: Bootstrap complete.
I20240503 02:00:20.894380 22226 ts_tablet_manager.cc:1367] T 02e8cbe02d1b4017909e9bfddaf093d8 P 63331f455be049c1bdd6c23c59ac1bef: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240503 02:00:20.894986 22226 raft_consensus.cc:348] T 02e8cbe02d1b4017909e9bfddaf093d8 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:20.895287 22226 raft_consensus.cc:729] T 02e8cbe02d1b4017909e9bfddaf093d8 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.895511 22226 consensus_queue.cc:260] T 02e8cbe02d1b4017909e9bfddaf093d8 P 63331f455be049c1bdd6c23c59ac1bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:20.896080 22126 tablet_bootstrap.cc:492] T 72e090a6c18e4a13bbb343eb40a07694 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:20.896509 22126 tablet_bootstrap.cc:492] T 72e090a6c18e4a13bbb343eb40a07694 P b7062fbbc2b54fff9e99ec4ea5562c76: Bootstrap complete.
I20240503 02:00:20.896061 22226 ts_tablet_manager.cc:1398] T 02e8cbe02d1b4017909e9bfddaf093d8 P 63331f455be049c1bdd6c23c59ac1bef: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:00:20.896704 22126 ts_tablet_manager.cc:1367] T 72e090a6c18e4a13bbb343eb40a07694 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent bootstrapping tablet: real 0.045s user 0.027s sys 0.000s
I20240503 02:00:20.897226 22126 raft_consensus.cc:348] T 72e090a6c18e4a13bbb343eb40a07694 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:20.897332 22126 raft_consensus.cc:729] T 72e090a6c18e4a13bbb343eb40a07694 P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7062fbbc2b54fff9e99ec4ea5562c76, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.897468 22126 consensus_queue.cc:260] T 72e090a6c18e4a13bbb343eb40a07694 P b7062fbbc2b54fff9e99ec4ea5562c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:20.897821 22126 ts_tablet_manager.cc:1398] T 72e090a6c18e4a13bbb343eb40a07694 P b7062fbbc2b54fff9e99ec4ea5562c76: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:20.898459 6903 tablet_copy_service.cc:342] P 5dcc33cc621a4cbbae155f3230fb7c46: Request end of tablet copy session b7062fbbc2b54fff9e99ec4ea5562c76-72e090a6c18e4a13bbb343eb40a07694 received from {username='slave'} at 127.0.99.135:54599
I20240503 02:00:20.898571 6903 tablet_copy_service.cc:433] P 5dcc33cc621a4cbbae155f3230fb7c46: ending tablet copy session b7062fbbc2b54fff9e99ec4ea5562c76-72e090a6c18e4a13bbb343eb40a07694 on tablet 72e090a6c18e4a13bbb343eb40a07694 with peer b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:20.899555 9383 tablet_copy_service.cc:342] P 12e8c683db054df3b75c6285b98a5ad8: Request end of tablet copy session 63331f455be049c1bdd6c23c59ac1bef-02e8cbe02d1b4017909e9bfddaf093d8 received from {username='slave'} at 127.0.99.130:52605
I20240503 02:00:20.899636 9383 tablet_copy_service.cc:433] P 12e8c683db054df3b75c6285b98a5ad8: ending tablet copy session 63331f455be049c1bdd6c23c59ac1bef-02e8cbe02d1b4017909e9bfddaf093d8 on tablet 02e8cbe02d1b4017909e9bfddaf093d8 with peer 63331f455be049c1bdd6c23c59ac1bef
I20240503 02:00:20.909310 22232 ts_tablet_manager.cc:902] T 822bafcbc6de406997955ff7a1d9b612 P 12e8c683db054df3b75c6285b98a5ad8: Initiating tablet copy from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 02:00:20.910156 22192 ts_tablet_manager.cc:902] T 9b9405669fb640c4a3a8fe2f31af291e P 1d0e738c34af40fa9950c6f086619b3a: Initiating tablet copy from peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 02:00:20.910737 22192 tablet_copy_client.cc:308] T 9b9405669fb640c4a3a8fe2f31af291e P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.129:42943
I20240503 02:00:20.911355 6903 tablet_copy_service.cc:140] P 5dcc33cc621a4cbbae155f3230fb7c46: Received BeginTabletCopySession request for tablet 9b9405669fb640c4a3a8fe2f31af291e from peer 1d0e738c34af40fa9950c6f086619b3a ({username='slave'} at 127.0.99.132:58605)
I20240503 02:00:20.911469 6903 tablet_copy_service.cc:161] P 5dcc33cc621a4cbbae155f3230fb7c46: Beginning new tablet copy session on tablet 9b9405669fb640c4a3a8fe2f31af291e from peer 1d0e738c34af40fa9950c6f086619b3a at {username='slave'} at 127.0.99.132:58605: session id = 1d0e738c34af40fa9950c6f086619b3a-9b9405669fb640c4a3a8fe2f31af291e
I20240503 02:00:20.912547 6903 tablet_copy_source_session.cc:205] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:20.912617 22239 ts_tablet_manager.cc:902] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Initiating tablet copy from peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 02:00:20.913631 22192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b9405669fb640c4a3a8fe2f31af291e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.913621 22239 tablet_copy_client.cc:308] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:43471
I20240503 02:00:20.916249 22192 tablet_copy_client.cc:783] T 9b9405669fb640c4a3a8fe2f31af291e P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:20.916417 22192 tablet_copy_client.cc:647] T 9b9405669fb640c4a3a8fe2f31af291e P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:20.916560 9514 tablet_copy_service.cc:140] P bb5364d730cb4489a0bfc6c270b7450a: Received BeginTabletCopySession request for tablet df13afe0f90c4c24b66026048b5852ec from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a ({username='slave'} at 127.0.99.131:52471)
I20240503 02:00:20.916664 9514 tablet_copy_service.cc:161] P bb5364d730cb4489a0bfc6c270b7450a: Beginning new tablet copy session on tablet df13afe0f90c4c24b66026048b5852ec from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a at {username='slave'} at 127.0.99.131:52471: session id = c688f3df56ac4b6d9b0c30fa4cc8ca2a-df13afe0f90c4c24b66026048b5852ec
I20240503 02:00:20.917683 9514 tablet_copy_source_session.cc:205] T df13afe0f90c4c24b66026048b5852ec P bb5364d730cb4489a0bfc6c270b7450a: Tablet Copy: opened 0 blocks and 1 log segments
F20240503 02:00:20.918341 22232 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-2065dcd6cde84f9a928e4b817737dd42.queue_time_us, tablet-download-pool-2065dcd6cde84f9a928e4b817737dd42.queue_time_us) (tablet-download-pool-2065dcd6cde84f9a928e4b817737dd42.run_cpu_time_us, tablet-download-pool-2065dcd6cde84f9a928e4b817737dd42.run_cpu_time_us) (tablet-download-pool-2065dcd6cde84f9a928e4b817737dd42.run_wall_time_us, tablet-download-pool-2065dcd6cde84f9a928e4b817737dd42.run_wall_time_us) (tablet-download-pool-38f126abb17c4654ae3cabe688d513d5.queue_time_us, tablet-download-pool-38f126abb17c4654ae3cabe688d513d5.queue_time_us) (tablet-download-pool-38f126abb17c4654ae3cabe688d513d5.run_cpu_time_us, tablet-download-pool-38f126abb17c4654ae3cabe688d513d5.run_cpu_time_us) (tablet-download-pool-38f126abb17c4654ae3cabe688d513d5.run_wall_time_us, tablet-download-pool-38f126abb17c4654ae3cabe688d513d5.run_wall_time_us) (tablet-download-pool-6d4480c8de204f5397dbdcff91db97f6.queue_time_us, tablet-download-pool-6d4480c8de204f5397dbdcff91db97f6.queue_time_us) (tablet-download-pool-6d4480c8de204f5397dbdcff91db97f6.run_cpu_time_us, tablet-download-pool-6d4480c8de204f5397dbdcff91db97f6.run_cpu_time_us) (tablet-download-pool-6d4480c8de204f5397dbdcff91db97f6.run_wall_time_us, tablet-download-pool-6d4480c8de204f5397dbdcff91db97f6.run_wall_time_us) (tablet-download-pool-72e090a6c18e4a13bbb343eb40a07694.queue_time_us, tablet-download-pool-72e090a6c18e4a13bbb343eb40a07694.queue_time_us) (tablet-download-pool-72e090a6c18e4a13bbb343eb40a07694.run_cpu_time_us, tablet-download-pool-72e090a6c18e4a13bbb343eb40a07694.run_cpu_time_us) (tablet-download-pool-72e090a6c18e4a13bbb343eb40a07694.run_wall_time_us, tablet-download-pool-72e090a6c18e4a13bbb343eb40a07694.run_wall_time_us) (tablet-download-pool-7672f42acd9446ef85eced07729c197b.queue_time_us, tablet-download-pool-7672f42acd9446ef85eced07729c197b.queue_time_us) (tablet-download-pool-7672f42acd9446ef85eced07729c197b.run_cpu_time_us, tablet-download-pool-7672f42acd9446ef85eced07729c197b.run_cpu_time_us) (tablet-download-pool-7672f42acd9446ef85eced07729c197b.run_wall_time_us, tablet-download-pool-7672f42acd9446ef85eced07729c197b.run_wall_time_us) (tablet-download-pool-822bafcbc6de406997955ff7a1d9b612.queue_time_us, tablet-download-pool-822bafcbc6de406997955ff7a1d9b612.queue_time_us) (tablet-download-pool-85d3e89fac21402387aa812f754cf5bc.queue_time_us, tablet-download-pool-85d3e89fac21402387aa812f754cf5bc.queue_time_us) (tablet-download-pool-85d3e89fac21402387aa812f754cf5bc.run_cpu_time_us, tablet-download-pool-85d3e89fac21402387aa812f754cf5bc.run_cpu_time_us) (tablet-download-pool-85d3e89fac21402387aa812f754cf5bc.run_wall_time_us, tablet-download-pool-85d3e89fac21402387aa812f754cf5bc.run_wall_time_us) (tablet-download-pool-9c7396d1b98c4b7ca495ba3ab5dcbbc6.queue_time_us, tablet-download-pool-9c7396d1b98c4b7ca495ba3ab5dcbbc6.queue_time_us) (tablet-download-pool-9c7396d1b98c4b7ca495ba3ab5dcbbc6.run_cpu_time_us, tablet-download-pool-9c7396d1b98c4b7ca495ba3ab5dcbbc6.run_cpu_time_us) (tablet-download-pool-9c7396d1b98c4b7ca495ba3ab5dcbbc6.run_wall_time_us, tablet-download-pool-9c7396d1b98c4b7ca495ba3ab5dcbbc6.run_wall_time_us) (tablet-download-pool-a5e23eebc629494cbb8262bc123a720a.queue_time_us, tablet-download-pool-a5e23eebc629494cbb8262bc123a720a.queue_time_us) (tablet-download-pool-a5e23eebc629494cbb8262bc123a720a.run_cpu_time_us, tablet-download-pool-a5e23eebc629494cbb8262bc123a720a.run_cpu_time_us) (tablet-download-pool-a5e23eebc629494cbb8262bc123a720a.run_wall_time_us, tablet-download-pool-a5e23eebc629494cbb8262bc123a720a.run_wall_time_us) (tablet-download-pool-c35ecf8890b64caaac7e5ea9f6e8f989.queue_time_us, tablet-download-pool-c35ecf8890b64caaac7e5ea9f6e8f989.queue_time_us) (tablet-download-pool-c35ecf8890b64caaac7e5ea9f6e8f989.run_cpu_time_us, tablet-download-pool-c35ecf8890b64caaac7e5ea9f6e8f989.run_cpu_time_us) (tablet-download-pool-c35ecf8890b64caaac7e5ea9f6e8f989.run_wall_time_us, tablet-download-pool-c35ecf8890b64caaac7e5ea9f6e8f989.run_wall_time_us) (tablet-download-pool-c4ac4ad7f9eb4723814f65ce96758c0e.queue_time_us, tablet-download-pool-c4ac4ad7f9eb4723814f65ce96758c0e.queue_time_us) (tablet-download-pool-c4ac4ad7f9eb4723814f65ce96758c0e.run_cpu_time_us, tablet-download-pool-c4ac4ad7f9eb4723814f65ce96758c0e.run_cpu_time_us) (tablet-download-pool-c4ac4ad7f9eb4723814f65ce96758c0e.run_wall_time_us, tablet-download-pool-c4ac4ad7f9eb4723814f65ce96758c0e.run_wall_time_us) (tablet-download-pool-ccbd932e20b44dc5aceb86c86d7fda6b.queue_time_us, tablet-download-pool-ccbd932e20b44dc5aceb86c86d7fda6b.queue_time_us) (tablet-download-pool-ccbd932e20b44dc5aceb86c86d7fda6b.run_cpu_time_us, tablet-download-pool-ccbd932e20b44dc5aceb86c86d7fda6b.run_cpu_time_us) (tablet-download-pool-ccbd932e20b44dc5aceb86c86d7fda6b.run_wall_time_us, tablet-download-pool-ccbd932e20b44dc5aceb86c86d7fda6b.run_wall_time_us) (tablet-download-pool-dcc773c6d5794e26a0003c7d4acc5d21.queue_time_us, tablet-download-pool-dcc773c6d5794e26a0003c7d4acc5d21.queue_time_us) (tablet-download-pool-dcc773c6d5794e26a0003c7d4acc5d21.run_cpu_time_us, tablet-download-pool-dcc773c6d5794e26a0003c7d4acc5d21.run_cpu_time_us) (tablet-download-pool-dcc773c6d5794e26a0003c7d4acc5d21.run_wall_time_us, tablet-download-pool-dcc773c6d5794e26a0003c7d4acc5d21.run_wall_time_us) (tablet-download-pool-e811662836164c3fb29a1b12ace7eb41.queue_time_us, tablet-download-pool-e811662836164c3fb29a1b12ace7eb41.queue_time_us) (tablet-download-pool-e811662836164c3fb29a1b12ace7eb41.run_cpu_time_us, tablet-download-pool-e811662836164c3fb29a1b12ace7eb41.run_cpu_time_us) (tablet-download-pool-e811662836164c3fb29a1b12ace7eb41.run_wall_time_us, tablet-download-pool-e811662836164c3fb29a1b12ace7eb41.run_wall_time_us) (tablet-download-pool-fd29f3b059874b019153b64ef8aaaa48.queue_time_us, tablet-download-pool-fd29f3b059874b019153b64ef8aaaa48.queue_time_us) (tablet-download-pool-fd29f3b059874b019153b64ef8aaaa48.run_cpu_time_us, tablet-download-pool-fd29f3b059874b019153b64ef8aaaa48.run_cpu_time_us) (tablet-download-pool-fd29f3b059874b019153b64ef8aaaa48.run_wall_time_us, tablet-download-pool-fd29f3b059874b019153b64ef8aaaa48.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240503 02:00:20.923760 22192 tablet_copy_client.cc:523] T 9b9405669fb640c4a3a8fe2f31af291e P 1d0e738c34af40fa9950c6f086619b3a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:20.924131 22239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df13afe0f90c4c24b66026048b5852ec. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f8cc2295dcd google::LogMessage::Fail() at ??:0
I20240503 02:00:20.927026 22192 tablet_bootstrap.cc:492] T 9b9405669fb640c4a3a8fe2f31af291e P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap starting.
@ 0x7f8cc2299b93 google::LogMessage::SendToLog() at ??:0
I20240503 02:00:20.928095 22239 tablet_copy_client.cc:783] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:20.928253 22198 ts_tablet_manager.cc:902] T 4600418bf64c4a0d962831c6924c46e1 P bb5364d730cb4489a0bfc6c270b7450a: Initiating tablet copy from peer 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 02:00:20.928273 22239 tablet_copy_client.cc:647] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Starting download of 1 WAL segments...
F20240503 02:00:20.928457 22198 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-2084fee295274ad89b407a933fb6070c.queue_time_us, tablet-download-pool-2084fee295274ad89b407a933fb6070c.queue_time_us) (tablet-download-pool-2084fee295274ad89b407a933fb6070c.run_cpu_time_us, tablet-download-pool-2084fee295274ad89b407a933fb6070c.run_cpu_time_us) (tablet-download-pool-2084fee295274ad89b407a933fb6070c.run_wall_time_us, tablet-download-pool-2084fee295274ad89b407a933fb6070c.run_wall_time_us) (tablet-download-pool-4600418bf64c4a0d962831c6924c46e1.queue_time_us, tablet-download-pool-4600418bf64c4a0d962831c6924c46e1.queue_time_us) (tablet-download-pool-487eafe47df84b468618f473eca37874.queue_time_us, tablet-download-pool-487eafe47df84b468618f473eca37874.queue_time_us) (tablet-download-pool-487eafe47df84b468618f473eca37874.run_cpu_time_us, tablet-download-pool-487eafe47df84b468618f473eca37874.run_cpu_time_us) (tablet-download-pool-487eafe47df84b468618f473eca37874.run_wall_time_us, tablet-download-pool-487eafe47df84b468618f473eca37874.run_wall_time_us) (tablet-download-pool-76e8697ac8624e03a46bddae35f4c9fa.queue_time_us, tablet-download-pool-76e8697ac8624e03a46bddae35f4c9fa.queue_time_us) (tablet-download-pool-76e8697ac8624e03a46bddae35f4c9fa.run_cpu_time_us, tablet-download-pool-76e8697ac8624e03a46bddae35f4c9fa.run_cpu_time_us) (tablet-download-pool-76e8697ac8624e03a46bddae35f4c9fa.run_wall_time_us, tablet-download-pool-76e8697ac8624e03a46bddae35f4c9fa.run_wall_time_us) (tablet-download-pool-822bafcbc6de406997955ff7a1d9b612.queue_time_us, tablet-download-pool-822bafcbc6de406997955ff7a1d9b612.queue_time_us) (tablet-download-pool-822bafcbc6de406997955ff7a1d9b612.run_cpu_time_us, tablet-download-pool-822bafcbc6de406997955ff7a1d9b612.run_cpu_time_us) (tablet-download-pool-822bafcbc6de406997955ff7a1d9b612.run_wall_time_us, tablet-download-pool-822bafcbc6de406997955ff7a1d9b612.run_wall_time_us) (tablet-download-pool-836882aa1fac4721be3cf096540eedc4.queue_time_us, tablet-download-pool-836882aa1fac4721be3cf096540eedc4.queue_time_us) (tablet-download-pool-836882aa1fac4721be3cf096540eedc4.run_cpu_time_us, tablet-download-pool-836882aa1fac4721be3cf096540eedc4.run_cpu_time_us) (tablet-download-pool-836882aa1fac4721be3cf096540eedc4.run_wall_time_us, tablet-download-pool-836882aa1fac4721be3cf096540eedc4.run_wall_time_us) (tablet-download-pool-87652406a8e442e58fe636a85ab72158.queue_time_us, tablet-download-pool-87652406a8e442e58fe636a85ab72158.queue_time_us) (tablet-download-pool-87652406a8e442e58fe636a85ab72158.run_cpu_time_us, tablet-download-pool-87652406a8e442e58fe636a85ab72158.run_cpu_time_us) (tablet-download-pool-87652406a8e442e58fe636a85ab72158.run_wall_time_us, tablet-download-pool-87652406a8e442e58fe636a85ab72158.run_wall_time_us) (tablet-download-pool-8ad236ea5ab347c69026293bef7c04a8.queue_time_us, tablet-download-pool-8ad236ea5ab347c69026293bef7c04a8.queue_time_us) (tablet-download-pool-8ad236ea5ab347c69026293bef7c04a8.run_cpu_time_us, tablet-download-pool-8ad236ea5ab347c69026293bef7c04a8.run_cpu_time_us) (tablet-download-pool-8ad236ea5ab347c69026293bef7c04a8.run_wall_time_us, tablet-download-pool-8ad236ea5ab347c69026293bef7c04a8.run_wall_time_us) (tablet-download-pool-8ed2793842464443826f83f60fb6dcd1.queue_time_us, tablet-download-pool-8ed2793842464443826f83f60fb6dcd1.queue_time_us) (tablet-download-pool-8ed2793842464443826f83f60fb6dcd1.run_cpu_time_us, tablet-download-pool-8ed2793842464443826f83f60fb6dcd1.run_cpu_time_us) (tablet-download-pool-8ed2793842464443826f83f60fb6dcd1.run_wall_time_us, tablet-download-pool-8ed2793842464443826f83f60fb6dcd1.run_wall_time_us) (tablet-download-pool-96dd9b1f85cf471495dff16d5fa9148a.queue_time_us, tablet-download-pool-96dd9b1f85cf471495dff16d5fa9148a.queue_time_us) (tablet-download-pool-96dd9b1f85cf471495dff16d5fa9148a.run_cpu_time_us, tablet-download-pool-96dd9b1f85cf471495dff16d5fa9148a.run_cpu_time_us) (tablet-download-pool-96dd9b1f85cf471495dff16d5fa9148a.run_wall_time_us, tablet-download-pool-96dd9b1f85cf471495dff16d5fa9148a.run_wall_time_us) (tablet-download-pool-c4ac4ad7f9eb4723814f65ce96758c0e.queue_time_us, tablet-download-pool-c4ac4ad7f9eb4723814f65ce96758c0e.queue_time_us) (tablet-download-pool-c4ac4ad7f9eb4723814f65ce96758c0e.run_cpu_time_us, tablet-download-pool-c4ac4ad7f9eb4723814f65ce96758c0e.run_cpu_time_us) (tablet-download-pool-c4ac4ad7f9eb4723814f65ce96758c0e.run_wall_time_us, tablet-download-pool-c4ac4ad7f9eb4723814f65ce96758c0e.run_wall_time_us) (tablet-download-pool-d80398c9a264430197262bb42241d58d.queue_time_us, tablet-download-pool-d80398c9a264430197262bb42241d58d.queue_time_us) (tablet-download-pool-d80398c9a264430197262bb42241d58d.run_cpu_time_us, tablet-download-pool-d80398c9a264430197262bb42241d58d.run_cpu_time_us) (tablet-download-pool-d80398c9a264430197262bb42241d58d.run_wall_time_us, tablet-download-pool-d80398c9a264430197262bb42241d58d.run_wall_time_us) (tablet-download-pool-db656aaa7b104b3182350de320f67839.queue_time_us, tablet-download-pool-db656aaa7b104b3182350de320f67839.queue_time_us) (tablet-download-pool-db656aaa7b104b3182350de320f67839.run_cpu_time_us, tablet-download-pool-db656aaa7b104b3182350de320f67839.run_cpu_time_us) (tablet-download-pool-db656aaa7b104b3182350de320f67839.run_wall_time_us, tablet-download-pool-db656aaa7b104b3182350de320f67839.run_wall_time_us) (tablet-download-pool-e811662836164c3fb29a1b12ace7eb41.queue_time_us, tablet-download-pool-e811662836164c3fb29a1b12ace7eb41.queue_time_us) (tablet-download-pool-e811662836164c3fb29a1b12ace7eb41.run_cpu_time_us, tablet-download-pool-e811662836164c3fb29a1b12ace7eb41.run_cpu_time_us) (tablet-download-pool-e811662836164c3fb29a1b12ace7eb41.run_wall_time_us, tablet-download-pool-e811662836164c3fb29a1b12ace7eb41.run_wall_time_us) (tablet-download-pool-fb287629baab4cada193ae57695ef1b5.queue_time_us, tablet-download-pool-fb287629baab4cada193ae57695ef1b5.queue_time_us) (tablet-download-pool-fb287629baab4cada193ae57695ef1b5.run_cpu_time_us, tablet-download-pool-fb287629baab4cada193ae57695ef1b5.run_cpu_time_us) (tablet-download-pool-fb287629baab4cada193ae57695ef1b5.run_wall_time_us, tablet-download-pool-fb287629baab4cada193ae57695ef1b5.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7fdcdebfedcd google::LogMessage::Fail() at ??:0
@ 0x7f8cc22957cc google::LogMessage::Flush() at ??:0
@ 0x7fdcdec02b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f8cc2296f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240503 02:00:20.936856 22239 tablet_copy_client.cc:523] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7fdcdebfe7cc google::LogMessage::Flush() at ??:0
I20240503 02:00:20.938684 22239 tablet_bootstrap.cc:492] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
@ 0x7f8cc36ff044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fdcdebfff59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8cc36e3f98 kudu::ThreadPool::ThreadPool() at ??:0
I20240503 02:00:20.945730 22192 tablet_bootstrap.cc:492] T 9b9405669fb640c4a3a8fe2f31af291e P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:20.946036 22192 tablet_bootstrap.cc:492] T 9b9405669fb640c4a3a8fe2f31af291e P 1d0e738c34af40fa9950c6f086619b3a: Bootstrap complete.
I20240503 02:00:20.946221 22192 ts_tablet_manager.cc:1367] T 9b9405669fb640c4a3a8fe2f31af291e P 1d0e738c34af40fa9950c6f086619b3a: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.001s
I20240503 02:00:20.946828 22192 raft_consensus.cc:348] T 9b9405669fb640c4a3a8fe2f31af291e P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:20.946962 22192 raft_consensus.cc:729] T 9b9405669fb640c4a3a8fe2f31af291e P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.947079 22192 consensus_queue.cc:260] T 9b9405669fb640c4a3a8fe2f31af291e P 1d0e738c34af40fa9950c6f086619b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:20.947376 22192 ts_tablet_manager.cc:1398] T 9b9405669fb640c4a3a8fe2f31af291e P 1d0e738c34af40fa9950c6f086619b3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 02:00:20.947892 6903 tablet_copy_service.cc:342] P 5dcc33cc621a4cbbae155f3230fb7c46: Request end of tablet copy session 1d0e738c34af40fa9950c6f086619b3a-9b9405669fb640c4a3a8fe2f31af291e received from {username='slave'} at 127.0.99.132:58605
I20240503 02:00:20.947970 6903 tablet_copy_service.cc:433] P 5dcc33cc621a4cbbae155f3230fb7c46: ending tablet copy session 1d0e738c34af40fa9950c6f086619b3a-9b9405669fb640c4a3a8fe2f31af291e on tablet 9b9405669fb640c4a3a8fe2f31af291e with peer 1d0e738c34af40fa9950c6f086619b3a
@ 0x7f8cc36e1efc kudu::ThreadPoolBuilder::Build() at ??:0
I20240503 02:00:20.955329 22248 ts_tablet_manager.cc:902] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46: Initiating tablet copy from peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 02:00:20.957381 22248 tablet_copy_client.cc:308] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.134:43471
I20240503 02:00:20.958835 9514 tablet_copy_service.cc:140] P bb5364d730cb4489a0bfc6c270b7450a: Received BeginTabletCopySession request for tablet 49eabbb8972840759ffa83951cb0b3b7 from peer 5dcc33cc621a4cbbae155f3230fb7c46 ({username='slave'} at 127.0.99.129:38243)
I20240503 02:00:20.958974 9514 tablet_copy_service.cc:161] P bb5364d730cb4489a0bfc6c270b7450a: Beginning new tablet copy session on tablet 49eabbb8972840759ffa83951cb0b3b7 from peer 5dcc33cc621a4cbbae155f3230fb7c46 at {username='slave'} at 127.0.99.129:38243: session id = 5dcc33cc621a4cbbae155f3230fb7c46-49eabbb8972840759ffa83951cb0b3b7
@ 0x7fdce0068044 kudu::TraceMetrics::InternName() at ??:0
I20240503 02:00:20.960494 9514 tablet_copy_source_session.cc:205] T 49eabbb8972840759ffa83951cb0b3b7 P bb5364d730cb4489a0bfc6c270b7450a: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:20.962253 22239 tablet_bootstrap.cc:492] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:20.962621 22239 tablet_bootstrap.cc:492] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap complete.
@ 0x7f8cc629477a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240503 02:00:20.962823 22239 ts_tablet_manager.cc:1367] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.001s
I20240503 02:00:20.963390 22239 raft_consensus.cc:348] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:20.963522 22239 raft_consensus.cc:729] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 02:00:20.964015 22239 consensus_queue.cc:260] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:20.964346 22239 ts_tablet_manager.cc:1398] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 02:00:20.965047 22248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49eabbb8972840759ffa83951cb0b3b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:20.966022 9514 tablet_copy_service.cc:342] P bb5364d730cb4489a0bfc6c270b7450a: Request end of tablet copy session c688f3df56ac4b6d9b0c30fa4cc8ca2a-df13afe0f90c4c24b66026048b5852ec received from {username='slave'} at 127.0.99.131:52471
I20240503 02:00:20.966111 9514 tablet_copy_service.cc:433] P bb5364d730cb4489a0bfc6c270b7450a: ending tablet copy session c688f3df56ac4b6d9b0c30fa4cc8ca2a-df13afe0f90c4c24b66026048b5852ec on tablet df13afe0f90c4c24b66026048b5852ec with peer c688f3df56ac4b6d9b0c30fa4cc8ca2a
@ 0x7f8cc629da45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f8cc631e76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 02:00:20.974280 22248 tablet_copy_client.cc:783] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:20.974483 22248 tablet_copy_client.cc:647] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Starting download of 1 WAL segments...
@ 0x7f8cc63324f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fdce004cf98 kudu::ThreadPool::ThreadPool() at ??:0
I20240503 02:00:20.978960 22248 tablet_copy_client.cc:523] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7f8cc631d226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240503 02:00:20.986840 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
@ 0x7f8cc632cc0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240503 02:00:20.988056 22248 tablet_bootstrap.cc:492] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap starting.
@ 0x5629c71bb75c std::function<>::operator()() at ??:0
@ 0x7fdce004aefc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fdce2bfd77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fdce2c06a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240503 02:00:21.002766 22142 tablet_bootstrap.cc:492] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:21.003078 22142 tablet_bootstrap.cc:492] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap complete.
I20240503 02:00:21.003232 22142 ts_tablet_manager.cc:1367] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.121s user 0.003s sys 0.002s
I20240503 02:00:21.004736 22248 tablet_bootstrap.cc:492] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:21.005010 22248 tablet_bootstrap.cc:492] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46: Bootstrap complete.
I20240503 02:00:21.005170 22248 ts_tablet_manager.cc:1367] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.004s
I20240503 02:00:21.005388 22142 raft_consensus.cc:348] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:21.005713 22142 raft_consensus.cc:729] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 02:00:21.005674 22248 raft_consensus.cc:348] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:21.005787 22248 raft_consensus.cc:729] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Initialized, Role: FOLLOWER
I20240503 02:00:21.005931 22142 consensus_queue.cc:260] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:21.005941 22248 consensus_queue.cc:260] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:21.006492 22142 ts_tablet_manager.cc:1398] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:00:21.007999 9514 tablet_copy_service.cc:342] P bb5364d730cb4489a0bfc6c270b7450a: Request end of tablet copy session 5dcc33cc621a4cbbae155f3230fb7c46-6eeb7c5fa08c48c08c36bf03bd1c85ae received from {username='slave'} at 127.0.99.129:38243
I20240503 02:00:21.008112 9514 tablet_copy_service.cc:433] P bb5364d730cb4489a0bfc6c270b7450a: ending tablet copy session 5dcc33cc621a4cbbae155f3230fb7c46-6eeb7c5fa08c48c08c36bf03bd1c85ae on tablet 6eeb7c5fa08c48c08c36bf03bd1c85ae with peer 5dcc33cc621a4cbbae155f3230fb7c46
I20240503 02:00:21.008469 22248 ts_tablet_manager.cc:1398] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
@ 0x7fdce2c8776f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 02:00:21.010998 9514 tablet_copy_service.cc:342] P bb5364d730cb4489a0bfc6c270b7450a: Request end of tablet copy session 5dcc33cc621a4cbbae155f3230fb7c46-49eabbb8972840759ffa83951cb0b3b7 received from {username='slave'} at 127.0.99.129:38243
I20240503 02:00:21.011209 9514 tablet_copy_service.cc:433] P bb5364d730cb4489a0bfc6c270b7450a: ending tablet copy session 5dcc33cc621a4cbbae155f3230fb7c46-49eabbb8972840759ffa83951cb0b3b7 on tablet 49eabbb8972840759ffa83951cb0b3b7 with peer 5dcc33cc621a4cbbae155f3230fb7c46
@ 0x7f8cc36e6e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fdce2c9b4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fdce2c86226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7fdce2c95c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55f5d125d75c std::function<>::operator()() at ??:0
@ 0x7f8cc36e76ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
I20240503 02:00:21.031832 22257 ts_tablet_manager.cc:902] T c43f13a33fe94033bce9debbae6990c2 P db1a0000aaa346f79c0aa4b6c5105191: Initiating tablet copy from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
@ 0x7fdce004fe06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fdce00506ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
I20240503 02:00:21.032491 22257 tablet_copy_client.cc:308] T c43f13a33fe94033bce9debbae6990c2 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.131:41979
I20240503 02:00:21.036727 7165 tablet_copy_service.cc:140] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Received BeginTabletCopySession request for tablet c43f13a33fe94033bce9debbae6990c2 from peer db1a0000aaa346f79c0aa4b6c5105191 ({username='slave'} at 127.0.99.136:40777)
I20240503 02:00:21.036859 7165 tablet_copy_service.cc:161] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Beginning new tablet copy session on tablet c43f13a33fe94033bce9debbae6990c2 from peer db1a0000aaa346f79c0aa4b6c5105191 at {username='slave'} at 127.0.99.136:40777: session id = db1a0000aaa346f79c0aa4b6c5105191-c43f13a33fe94033bce9debbae6990c2
I20240503 02:00:21.038233 7165 tablet_copy_source_session.cc:205] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Tablet Copy: opened 0 blocks and 1 log segments
@ 0x7f8cc36e903b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x5629c71bb75c std::function<>::operator()() at ??:0
@ 0x7fdce005203b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55f5d125d75c std::function<>::operator()() at ??:0
I20240503 02:00:21.041218 22257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c43f13a33fe94033bce9debbae6990c2. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f8cc36d6807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdce003f807 kudu::Thread::SuperviseThread() at ??:0
I20240503 02:00:21.046005 22257 tablet_copy_client.cc:783] T c43f13a33fe94033bce9debbae6990c2 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Starting download of 0 data blocks...
@ 0x7f8cc3f146db start_thread at ??:0
I20240503 02:00:21.046258 22257 tablet_copy_client.cc:647] T c43f13a33fe94033bce9debbae6990c2 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Starting download of 1 WAL segments...
@ 0x7fdce087d6db start_thread at ??:0
@ 0x7f8cc144c71f clone at ??:0
I20240503 02:00:21.052830 22257 tablet_copy_client.cc:523] T c43f13a33fe94033bce9debbae6990c2 P db1a0000aaa346f79c0aa4b6c5105191: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7fdcdddb571f clone at ??:0
I20240503 02:00:21.055609 22257 tablet_bootstrap.cc:492] T c43f13a33fe94033bce9debbae6990c2 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap starting.
I20240503 02:00:21.063863 22257 tablet_bootstrap.cc:492] T c43f13a33fe94033bce9debbae6990c2 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:00:21.064198 22257 tablet_bootstrap.cc:492] T c43f13a33fe94033bce9debbae6990c2 P db1a0000aaa346f79c0aa4b6c5105191: Bootstrap complete.
I20240503 02:00:21.064517 22257 ts_tablet_manager.cc:1367] T c43f13a33fe94033bce9debbae6990c2 P db1a0000aaa346f79c0aa4b6c5105191: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20240503 02:00:21.065104 22257 raft_consensus.cc:348] T c43f13a33fe94033bce9debbae6990c2 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:21.065234 22257 raft_consensus.cc:729] T c43f13a33fe94033bce9debbae6990c2 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Initialized, Role: FOLLOWER
I20240503 02:00:21.065426 22257 consensus_queue.cc:260] T c43f13a33fe94033bce9debbae6990c2 P db1a0000aaa346f79c0aa4b6c5105191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:21.066184 22257 ts_tablet_manager.cc:1398] T c43f13a33fe94033bce9debbae6990c2 P db1a0000aaa346f79c0aa4b6c5105191: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:00:21.066839 7165 tablet_copy_service.cc:342] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Request end of tablet copy session db1a0000aaa346f79c0aa4b6c5105191-c43f13a33fe94033bce9debbae6990c2 received from {username='slave'} at 127.0.99.136:40777
I20240503 02:00:21.066927 7165 tablet_copy_service.cc:433] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: ending tablet copy session db1a0000aaa346f79c0aa4b6c5105191-c43f13a33fe94033bce9debbae6990c2 on tablet c43f13a33fe94033bce9debbae6990c2 with peer db1a0000aaa346f79c0aa4b6c5105191
I20240503 02:00:21.078285 22126 ts_tablet_manager.cc:902] T c6833c997d284063955d0b9b09ccb3b7 P b7062fbbc2b54fff9e99ec4ea5562c76: Initiating tablet copy from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
F20240503 02:00:21.078538 22126 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-22b6ec13c4e04d6cbade150f0c644e52.queue_time_us, tablet-download-pool-22b6ec13c4e04d6cbade150f0c644e52.queue_time_us) (tablet-download-pool-22b6ec13c4e04d6cbade150f0c644e52.run_cpu_time_us, tablet-download-pool-22b6ec13c4e04d6cbade150f0c644e52.run_cpu_time_us) (tablet-download-pool-22b6ec13c4e04d6cbade150f0c644e52.run_wall_time_us, tablet-download-pool-22b6ec13c4e04d6cbade150f0c644e52.run_wall_time_us) (tablet-download-pool-49b580d648014ce899fee8b3b329c47d.queue_time_us, tablet-download-pool-49b580d648014ce899fee8b3b329c47d.queue_time_us) (tablet-download-pool-49b580d648014ce899fee8b3b329c47d.run_cpu_time_us, tablet-download-pool-49b580d648014ce899fee8b3b329c47d.run_cpu_time_us) (tablet-download-pool-49b580d648014ce899fee8b3b329c47d.run_wall_time_us, tablet-download-pool-49b580d648014ce899fee8b3b329c47d.run_wall_time_us) (tablet-download-pool-4bd8226227c348d5a5d4d8ff28959d27.queue_time_us, tablet-download-pool-4bd8226227c348d5a5d4d8ff28959d27.queue_time_us) (tablet-download-pool-4bd8226227c348d5a5d4d8ff28959d27.run_cpu_time_us, tablet-download-pool-4bd8226227c348d5a5d4d8ff28959d27.run_cpu_time_us) (tablet-download-pool-4bd8226227c348d5a5d4d8ff28959d27.run_wall_time_us, tablet-download-pool-4bd8226227c348d5a5d4d8ff28959d27.run_wall_time_us) (tablet-download-pool-580e6b4d2b2f4f79b8c2af5e75a3c75f.queue_time_us, tablet-download-pool-580e6b4d2b2f4f79b8c2af5e75a3c75f.queue_time_us) (tablet-download-pool-580e6b4d2b2f4f79b8c2af5e75a3c75f.run_cpu_time_us, tablet-download-pool-580e6b4d2b2f4f79b8c2af5e75a3c75f.run_cpu_time_us) (tablet-download-pool-580e6b4d2b2f4f79b8c2af5e75a3c75f.run_wall_time_us, tablet-download-pool-580e6b4d2b2f4f79b8c2af5e75a3c75f.run_wall_time_us) (tablet-download-pool-5cb5072e91534671ad3b4dd85f93fa5b.queue_time_us, tablet-download-pool-5cb5072e91534671ad3b4dd85f93fa5b.queue_time_us) (tablet-download-pool-5cb5072e91534671ad3b4dd85f93fa5b.run_cpu_time_us, tablet-download-pool-5cb5072e91534671ad3b4dd85f93fa5b.run_cpu_time_us) (tablet-download-pool-5cb5072e91534671ad3b4dd85f93fa5b.run_wall_time_us, tablet-download-pool-5cb5072e91534671ad3b4dd85f93fa5b.run_wall_time_us) (tablet-download-pool-72e090a6c18e4a13bbb343eb40a07694.queue_time_us, tablet-download-pool-72e090a6c18e4a13bbb343eb40a07694.queue_time_us) (tablet-download-pool-72e090a6c18e4a13bbb343eb40a07694.run_cpu_time_us, tablet-download-pool-72e090a6c18e4a13bbb343eb40a07694.run_cpu_time_us) (tablet-download-pool-72e090a6c18e4a13bbb343eb40a07694.run_wall_time_us, tablet-download-pool-72e090a6c18e4a13bbb343eb40a07694.run_wall_time_us) (tablet-download-pool-8295ea038a88485095544cc608b000c2.queue_time_us, tablet-download-pool-8295ea038a88485095544cc608b000c2.queue_time_us) (tablet-download-pool-8295ea038a88485095544cc608b000c2.run_cpu_time_us, tablet-download-pool-8295ea038a88485095544cc608b000c2.run_cpu_time_us) (tablet-download-pool-8295ea038a88485095544cc608b000c2.run_wall_time_us, tablet-download-pool-8295ea038a88485095544cc608b000c2.run_wall_time_us) (tablet-download-pool-85d3e89fac21402387aa812f754cf5bc.queue_time_us, tablet-download-pool-85d3e89fac21402387aa812f754cf5bc.queue_time_us) (tablet-download-pool-85d3e89fac21402387aa812f754cf5bc.run_cpu_time_us, tablet-download-pool-85d3e89fac21402387aa812f754cf5bc.run_cpu_time_us) (tablet-download-pool-85d3e89fac21402387aa812f754cf5bc.run_wall_time_us, tablet-download-pool-85d3e89fac21402387aa812f754cf5bc.run_wall_time_us) (tablet-download-pool-8c3d43c8b2ca42b19616fde1d7eb47a3.queue_time_us, tablet-download-pool-8c3d43c8b2ca42b19616fde1d7eb47a3.queue_time_us) (tablet-download-pool-8c3d43c8b2ca42b19616fde1d7eb47a3.run_cpu_time_us, tablet-download-pool-8c3d43c8b2ca42b19616fde1d7eb47a3.run_cpu_time_us) (tablet-download-pool-8c3d43c8b2ca42b19616fde1d7eb47a3.run_wall_time_us, tablet-download-pool-8c3d43c8b2ca42b19616fde1d7eb47a3.run_wall_time_us) (tablet-download-pool-c029253a832e46a48ae16691fc044502.queue_time_us, tablet-download-pool-c029253a832e46a48ae16691fc044502.queue_time_us) (tablet-download-pool-c029253a832e46a48ae16691fc044502.run_cpu_time_us, tablet-download-pool-c029253a832e46a48ae16691fc044502.run_cpu_time_us) (tablet-download-pool-c029253a832e46a48ae16691fc044502.run_wall_time_us, tablet-download-pool-c029253a832e46a48ae16691fc044502.run_wall_time_us) (tablet-download-pool-c4930b3ca84749f6a4cd245411f66075.queue_time_us, tablet-download-pool-c4930b3ca84749f6a4cd245411f66075.queue_time_us) (tablet-download-pool-c4930b3ca84749f6a4cd245411f66075.run_cpu_time_us, tablet-download-pool-c4930b3ca84749f6a4cd245411f66075.run_cpu_time_us) (tablet-download-pool-c4930b3ca84749f6a4cd245411f66075.run_wall_time_us, tablet-download-pool-c4930b3ca84749f6a4cd245411f66075.run_wall_time_us) (tablet-download-pool-c6833c997d284063955d0b9b09ccb3b7.queue_time_us, tablet-download-pool-c6833c997d284063955d0b9b09ccb3b7.queue_time_us) (tablet-download-pool-d2fbadb911aa4bd7b4db021860c202d3.queue_time_us, tablet-download-pool-d2fbadb911aa4bd7b4db021860c202d3.queue_time_us) (tablet-download-pool-d2fbadb911aa4bd7b4db021860c202d3.run_cpu_time_us, tablet-download-pool-d2fbadb911aa4bd7b4db021860c202d3.run_cpu_time_us) (tablet-download-pool-d2fbadb911aa4bd7b4db021860c202d3.run_wall_time_us, tablet-download-pool-d2fbadb911aa4bd7b4db021860c202d3.run_wall_time_us) (tablet-download-pool-dec7e3036d384f5db51f50ba93acbbc6.queue_time_us, tablet-download-pool-dec7e3036d384f5db51f50ba93acbbc6.queue_time_us) (tablet-download-pool-dec7e3036d384f5db51f50ba93acbbc6.run_cpu_time_us, tablet-download-pool-dec7e3036d384f5db51f50ba93acbbc6.run_cpu_time_us) (tablet-download-pool-dec7e3036d384f5db51f50ba93acbbc6.run_wall_time_us, tablet-download-pool-dec7e3036d384f5db51f50ba93acbbc6.run_wall_time_us) (tablet-download-pool-dfdcdd54606e4320a5d0f506eb6dabf8.queue_time_us, tablet-download-pool-dfdcdd54606e4320a5d0f506eb6dabf8.queue_time_us) (tablet-download-pool-dfdcdd54606e4320a5d0f506eb6dabf8.run_cpu_time_us, tablet-download-pool-dfdcdd54606e4320a5d0f506eb6dabf8.run_cpu_time_us) (tablet-download-pool-dfdcdd54606e4320a5d0f506eb6dabf8.run_wall_time_us, tablet-download-pool-dfdcdd54606e4320a5d0f506eb6dabf8.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240503 02:00:21.084540 9624 raft_consensus.cc:1203] T 85d3e89fac21402387aa812f754cf5bc P b7062fbbc2b54fff9e99ec4ea5562c76 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
@ 0x7f488bb50dcd google::LogMessage::Fail() at ??:0
@ 0x7f488bb54b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f488bb507cc google::LogMessage::Flush() at ??:0
@ 0x7f488bb51f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f488cfba044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f488cf9ef98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f488cf9cefc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f488fb4f77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f488fb58a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f488fbd976f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f488fbed4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f488fbd8226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f488fbe7c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55690114b75c std::function<>::operator()() at ??:0
@ 0x7f488cfa1e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f488cfa26ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f488cfa403b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55690114b75c std::function<>::operator()() at ??:0
@ 0x7f488cf91807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f488d7cf6db start_thread at ??:0
@ 0x7f488ad0771f clone at ??:0
I20240503 02:00:21.204160 7274 raft_consensus.cc:1203] T 822cbc676b234eda8770f7b485ebe302 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:21.229362 6882 raft_consensus.cc:1203] T b5329f65c13c47d485cee81cf9cf4f1e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:21.496558 7274 raft_consensus.cc:1203] T 9b9405669fb640c4a3a8fe2f31af291e P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:00:21.525218 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:21.527964 6861 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:21.540010 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:21.557222 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:00:21.565428 6816 connection.cc:535] client connection to 127.0.99.134:43471 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:21.565394 6947 connection.cc:535] client connection to 127.0.99.134:43471 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:21.565578 6816 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:21.565578 6947 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:21.566612 7078 connection.cc:535] client connection to 127.0.99.134:43471 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:21.566738 7078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:21.567461 9690 connection.cc:535] client connection to 127.0.99.134:43471 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:21.567576 9690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:21.568011 7209 connection.cc:535] client connection to 127.0.99.134:43471 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:21.568116 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:00:21.573043 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 02:00:21.583662 22371 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:00:21.579531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.134:43471 (local address 127.0.99.132:39937)
0503 02:00:21.582326 (+ 2795us) negotiation.cc:107] Waiting for socket to connect
0503 02:00:21.582894 (+ 568us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":2646,"thread_start_us":141,"threads_started":1}
W20240503 02:00:21.589601 6816 consensus_peers.cc:485] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.590335 6816 consensus_peers.cc:485] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.591315 6816 consensus_peers.cc:485] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.591400 6816 consensus_peers.cc:485] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:21.612553 9756 raft_consensus.cc:1203] T c43f13a33fe94033bce9debbae6990c2 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240503 02:00:21.613830 6818 connection.cc:535] client connection to 127.0.99.133:37387 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:21.614028 7211 connection.cc:535] client connection to 127.0.99.133:37387 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:21.614215 7080 connection.cc:535] client connection to 127.0.99.133:37387 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:21.617722 6949 connection.cc:535] client connection to 127.0.99.133:37387 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:21.617796 9692 connection.cc:535] client connection to 127.0.99.133:37387 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:21.620518 22374 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:00:21.614858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.133:37387 (local address 127.0.99.129:60785)
0503 02:00:21.615019 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0503 02:00:21.615039 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0503 02:00:21.615801 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:00:21.618971 (+ 3170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":56}
W20240503 02:00:21.620644 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.620728 6818 consensus_peers.cc:485] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.620774 6818 consensus_peers.cc:485] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.620817 6818 consensus_peers.cc:485] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.620873 6818 consensus_peers.cc:485] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.620908 6818 consensus_peers.cc:485] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.620942 6818 consensus_peers.cc:485] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.620977 6818 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.621011 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.622645 22376 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:00:21.570335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.134:43471 (local address 127.0.99.131:46895)
0503 02:00:21.621876 (+ 51541us) negotiation.cc:107] Waiting for socket to connect
0503 02:00:21.621950 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":51465,"thread_start_us":77,"threads_started":1}
W20240503 02:00:21.622776 7078 consensus_peers.cc:485] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.622844 7078 consensus_peers.cc:485] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.622844 6947 consensus_peers.cc:485] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.622902 7078 consensus_peers.cc:485] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.622925 6947 consensus_peers.cc:485] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.622956 7078 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.622973 6947 consensus_peers.cc:485] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.623023 6947 consensus_peers.cc:485] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.623059 6947 consensus_peers.cc:485] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.623095 6947 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.623097 7078 consensus_peers.cc:485] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.623128 6947 consensus_peers.cc:485] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.623160 7078 consensus_peers.cc:485] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.623217 7078 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.623239 7080 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.623271 7078 consensus_peers.cc:485] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.623298 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:21.623334 7080 consensus_peers.cc:485] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.623382 7080 consensus_peers.cc:485] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.623411 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.623522 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.623574 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.623610 6949 consensus_peers.cc:485] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.623642 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.630453 7209 consensus_peers.cc:485] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.630594 7209 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.630643 7209 consensus_peers.cc:485] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.630683 7209 consensus_peers.cc:485] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.630717 7209 consensus_peers.cc:485] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.630765 7209 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.630810 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:21.630954 7211 consensus_peers.cc:485] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.631006 7211 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.636608 9690 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.636700 9690 consensus_peers.cc:485] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.636739 9690 consensus_peers.cc:485] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.636771 9690 consensus_peers.cc:485] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.636806 9690 consensus_peers.cc:485] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.637099 9692 consensus_peers.cc:485] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.637156 9692 consensus_peers.cc:485] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.637251 7211 consensus_peers.cc:485] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.637585 7078 consensus_peers.cc:485] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.638353 7211 consensus_peers.cc:485] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.638638 7078 consensus_peers.cc:485] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.641016 7211 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.697558 6818 connection.cc:535] client connection to 127.0.99.135:46087 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:21.697753 7080 connection.cc:535] client connection to 127.0.99.135:46087 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:21.697821 9692 connection.cc:535] client connection to 127.0.99.135:46087 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:21.697875 6949 connection.cc:535] client connection to 127.0.99.135:46087 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:21.699401 7211 connection.cc:535] client connection to 127.0.99.135:46087 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:21.700537 22379 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:00:21.700112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:46087 (local address 127.0.99.132:57467)
0503 02:00:21.700372 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0503 02:00:21.700440 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":146}
W20240503 02:00:21.700758 22382 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:00:21.698595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:46087 (local address 127.0.99.136:50727)
0503 02:00:21.698908 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0503 02:00:21.698927 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0503 02:00:21.699038 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:00:21.700157 (+ 1119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":198}
W20240503 02:00:21.701134 7211 consensus_peers.cc:485] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.701208 7211 consensus_peers.cc:485] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.701203 9692 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.701246 7211 consensus_peers.cc:485] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.701305 9692 consensus_peers.cc:485] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.701352 9692 consensus_peers.cc:485] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.701387 9692 consensus_peers.cc:485] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.701367 22373 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:00:21.699002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:46087 (local address 127.0.99.130:56891)
0503 02:00:21.699482 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0503 02:00:21.699500 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0503 02:00:21.699664 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:00:21.700903 (+ 1239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":365}
W20240503 02:00:21.701563 9692 consensus_peers.cc:485] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.701793 22370 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:00:21.699022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:46087 (local address 127.0.99.131:38791)
0503 02:00:21.699300 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0503 02:00:21.699315 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0503 02:00:21.699420 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:00:21.701664 (+ 2244us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":189}
W20240503 02:00:21.701812 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.701874 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.701936 6949 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.701961 7080 consensus_peers.cc:485] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.701977 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.702023 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:00:21.702021 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.702066 6949 consensus_peers.cc:485] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.702067 7080 consensus_peers.cc:485] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.702118 6949 consensus_peers.cc:485] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.702118 7080 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.702164 6949 consensus_peers.cc:485] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.702165 7080 consensus_peers.cc:485] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.702487 22374 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:00:21.699426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:46087 (local address 127.0.99.129:39067)
0503 02:00:21.699799 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0503 02:00:21.699811 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0503 02:00:21.699900 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:00:21.702393 (+ 2493us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":283}
W20240503 02:00:21.702804 6818 consensus_peers.cc:485] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.702867 6818 consensus_peers.cc:485] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.702904 6818 consensus_peers.cc:485] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.702937 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.702988 6818 consensus_peers.cc:485] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:21.703025 6818 consensus_peers.cc:485] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:21.884418 6861 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:21.885629 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:21.887257 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:21.890995 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:00:21.895068 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:00:21.903852 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:21.928376 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:22.022238 22255 raft_consensus.cc:482] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:22.022454 22255 raft_consensus.cc:504] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:22.023461 22255 leader_election.cc:288] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 02:00:22.024964 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49eabbb8972840759ffa83951cb0b3b7" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 02:00:22.025161 6882 raft_consensus.cc:2454] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
W20240503 02:00:22.025467 9692 leader_election.cc:334] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:22.030018 9690 leader_election.cc:334] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:22.030148 9690 leader_election.cc:302] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, db1a0000aaa346f79c0aa4b6c5105191; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:22.030498 22233 raft_consensus.cc:2726] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.053704 21176 raft_consensus.cc:482] T 5097893b51da40059b0ef865d42c4d4a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.053901 21176 raft_consensus.cc:504] T 5097893b51da40059b0ef865d42c4d4a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:22.054944 21176 leader_election.cc:288] T 5097893b51da40059b0ef865d42c4d4a P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 02:00:22.056269 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5097893b51da40059b0ef865d42c4d4a" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
W20240503 02:00:22.057140 6818 leader_election.cc:334] T 5097893b51da40059b0ef865d42c4d4a P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.057337 6817 leader_election.cc:302] T 5097893b51da40059b0ef865d42c4d4a P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, db1a0000aaa346f79c0aa4b6c5105191
I20240503 02:00:22.058678 21176 raft_consensus.cc:2726] T 5097893b51da40059b0ef865d42c4d4a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.084549 21721 raft_consensus.cc:482] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.084708 21721 raft_consensus.cc:504] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:22.085333 21721 leader_election.cc:288] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:22.089408 6949 leader_election.cc:334] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:22.090054 6949 leader_election.cc:334] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.090147 6949 leader_election.cc:302] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:22.090322 21721 raft_consensus.cc:2726] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.099448 22042 raft_consensus.cc:482] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:22.099622 22042 raft_consensus.cc:504] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:22.100541 22042 leader_election.cc:288] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:22.104218 7211 leader_election.cc:334] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.104216 21721 raft_consensus.cc:482] T 5ec2909a6e484f81aad7b39a374e1ef8 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:22.104362 21721 raft_consensus.cc:504] T 5ec2909a6e484f81aad7b39a374e1ef8 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:22.105118 21721 leader_election.cc:288] T 5ec2909a6e484f81aad7b39a374e1ef8 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 02:00:22.105688 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ec2909a6e484f81aad7b39a374e1ef8" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
W20240503 02:00:22.106817 7209 leader_election.cc:334] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:22.107064 7209 leader_election.cc:302] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
W20240503 02:00:22.107136 6947 leader_election.cc:334] T 5ec2909a6e484f81aad7b39a374e1ef8 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:22.107211 6947 leader_election.cc:302] T 5ec2909a6e484f81aad7b39a374e1ef8 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 1d0e738c34af40fa9950c6f086619b3a, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:22.107445 21721 raft_consensus.cc:2726] T 5ec2909a6e484f81aad7b39a374e1ef8 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.107445 22042 raft_consensus.cc:2726] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.118558 22255 raft_consensus.cc:482] T 6fd0378b2cff42a681361930f487d834 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.118719 22255 raft_consensus.cc:504] T 6fd0378b2cff42a681361930f487d834 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:22.119191 22255 leader_election.cc:288] T 6fd0378b2cff42a681361930f487d834 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 02:00:22.119812 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd0378b2cff42a681361930f487d834" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 02:00:22.120756 21721 raft_consensus.cc:482] T b93fe374d2a9496a847480b7824a5937 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.120873 21721 raft_consensus.cc:504] T b93fe374d2a9496a847480b7824a5937 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:22.121310 21721 leader_election.cc:288] T b93fe374d2a9496a847480b7824a5937 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:22.121861 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b93fe374d2a9496a847480b7824a5937" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
W20240503 02:00:22.122496 9692 leader_election.cc:334] T 6fd0378b2cff42a681361930f487d834 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.123239 9690 leader_election.cc:302] T 6fd0378b2cff42a681361930f487d834 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, 63331f455be049c1bdd6c23c59ac1bef
I20240503 02:00:22.123517 22255 raft_consensus.cc:2726] T 6fd0378b2cff42a681361930f487d834 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:22.125207 6949 leader_election.cc:334] T b93fe374d2a9496a847480b7824a5937 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:22.125375 6949 leader_election.cc:302] T b93fe374d2a9496a847480b7824a5937 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, db1a0000aaa346f79c0aa4b6c5105191
I20240503 02:00:22.125566 21721 raft_consensus.cc:2726] T b93fe374d2a9496a847480b7824a5937 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.137166 21176 raft_consensus.cc:482] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:22.137311 21176 raft_consensus.cc:504] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:22.137759 21176 leader_election.cc:288] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:22.139655 6816 leader_election.cc:334] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:22.139777 6818 leader_election.cc:334] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.139873 6818 leader_election.cc:302] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:22.140126 21176 raft_consensus.cc:2726] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.141997 21176 raft_consensus.cc:482] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:22.142112 21176 raft_consensus.cc:504] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:22.142699 21176 leader_election.cc:288] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 02:00:22.143340 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeb7c5fa08c48c08c36bf03bd1c85ae" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 02:00:22.143752 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeb7c5fa08c48c08c36bf03bd1c85ae" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 02:00:22.144420 6817 leader_election.cc:302] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 1d0e738c34af40fa9950c6f086619b3a, db1a0000aaa346f79c0aa4b6c5105191
I20240503 02:00:22.145185 21176 raft_consensus.cc:2726] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:22.151017 6816 leader_election.cc:334] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:22.154217 21292 raft_consensus.cc:482] T d1df18d990ee43a596a4656dd498b8e7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:22.154414 21292 raft_consensus.cc:504] T d1df18d990ee43a596a4656dd498b8e7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:22.154937 21292 leader_election.cc:288] T d1df18d990ee43a596a4656dd498b8e7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 02:00:22.155030 21292 raft_consensus.cc:482] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.155095 21292 raft_consensus.cc:504] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:22.155454 21292 leader_election.cc:288] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 02:00:22.155802 21540 raft_consensus.cc:482] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.155907 21540 raft_consensus.cc:504] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:22.156293 21540 leader_election.cc:288] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:22.156613 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1df18d990ee43a596a4656dd498b8e7" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
W20240503 02:00:22.160993 7080 leader_election.cc:334] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:22.161121 7080 leader_election.cc:334] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:22.161201 7080 leader_election.cc:334] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:22.161238 7080 leader_election.cc:302] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:22.161571 7254 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:00:22.162117 21540 raft_consensus.cc:2726] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:22.162322 7078 leader_election.cc:334] T d1df18d990ee43a596a4656dd498b8e7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:22.162393 7078 leader_election.cc:302] T d1df18d990ee43a596a4656dd498b8e7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: bb5364d730cb4489a0bfc6c270b7450a, db1a0000aaa346f79c0aa4b6c5105191
W20240503 02:00:22.162602 7078 leader_election.cc:334] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:22.162670 7078 leader_election.cc:302] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:22.163007 21540 raft_consensus.cc:2726] T d1df18d990ee43a596a4656dd498b8e7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.163127 21540 raft_consensus.cc:2726] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.170109 21292 raft_consensus.cc:482] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.170274 21292 raft_consensus.cc:504] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:22.170878 21292 leader_election.cc:288] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:22.171861 9735 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 02:00:22.173566 7080 leader_election.cc:334] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:22.174407 7080 leader_election.cc:334] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.174487 7080 leader_election.cc:302] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:22.174677 21292 raft_consensus.cc:2726] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.179077 7123 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:22.183324 22255 raft_consensus.cc:482] T 1d182ce2c5084082a0990ce8090eb88b P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.183458 22255 raft_consensus.cc:504] T 1d182ce2c5084082a0990ce8090eb88b P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:22.184497 22255 leader_election.cc:288] T 1d182ce2c5084082a0990ce8090eb88b P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
W20240503 02:00:22.189594 9692 leader_election.cc:334] T 1d182ce2c5084082a0990ce8090eb88b P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:22.189889 6861 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:22.190258 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d182ce2c5084082a0990ce8090eb88b" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 02:00:22.190841 9691 leader_election.cc:302] T 1d182ce2c5084082a0990ce8090eb88b P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 1d0e738c34af40fa9950c6f086619b3a, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:22.191112 22255 raft_consensus.cc:2726] T 1d182ce2c5084082a0990ce8090eb88b P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.196494 6992 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:22.199625 21292 raft_consensus.cc:482] T b886572c6ec9473bbec744cee8de12f5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.199769 21292 raft_consensus.cc:504] T b886572c6ec9473bbec744cee8de12f5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:22.200814 6883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b886572c6ec9473bbec744cee8de12f5" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 02:00:22.202229 21292 leader_election.cc:288] T b886572c6ec9473bbec744cee8de12f5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 02:00:22.209501 22255 raft_consensus.cc:482] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:22.209667 22255 raft_consensus.cc:504] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:22.210330 22255 leader_election.cc:288] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
W20240503 02:00:22.211942 9690 leader_election.cc:334] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:22.212436 7080 leader_election.cc:334] T b886572c6ec9473bbec744cee8de12f5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.212626 7080 leader_election.cc:302] T b886572c6ec9473bbec744cee8de12f5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, 5dcc33cc621a4cbbae155f3230fb7c46
I20240503 02:00:22.212868 21292 raft_consensus.cc:2726] T b886572c6ec9473bbec744cee8de12f5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.213317 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeb7c5fa08c48c08c36bf03bd1c85ae" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 02:00:22.213516 6882 raft_consensus.cc:2454] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 02:00:22.213869 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeb7c5fa08c48c08c36bf03bd1c85ae" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 02:00:22.214358 9691 leader_election.cc:302] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, db1a0000aaa346f79c0aa4b6c5105191; no voters: 1d0e738c34af40fa9950c6f086619b3a, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:22.214679 22255 raft_consensus.cc:2726] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.220465 22042 raft_consensus.cc:482] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:22.220616 22042 raft_consensus.cc:504] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:22.221176 22042 leader_election.cc:288] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 02:00:22.221719 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeb7c5fa08c48c08c36bf03bd1c85ae" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 02:00:22.222050 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeb7c5fa08c48c08c36bf03bd1c85ae" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 02:00:22.222103 6882 raft_consensus.cc:2454] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 02:00:22.222191 9756 raft_consensus.cc:2454] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 02:00:22.222730 7210 leader_election.cc:302] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46, db1a0000aaa346f79c0aa4b6c5105191; no voters:
W20240503 02:00:22.222883 7209 leader_election.cc:334] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:22.222949 22042 raft_consensus.cc:2781] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:00:22.223011 22042 raft_consensus.cc:482] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:22.223058 22042 raft_consensus.cc:3037] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.224411 22042 raft_consensus.cc:504] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:22.226414 22042 leader_election.cc:288] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 election: Requested vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
W20240503 02:00:22.226766 7209 leader_election.cc:334] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:22.227043 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeb7c5fa08c48c08c36bf03bd1c85ae" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 02:00:22.227195 6882 raft_consensus.cc:3037] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.227543 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eeb7c5fa08c48c08c36bf03bd1c85ae" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 02:00:22.227660 9756 raft_consensus.cc:3037] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.228617 6882 raft_consensus.cc:2454] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 5dcc33cc621a4cbbae155f3230fb7c46 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 2.
I20240503 02:00:22.228863 9756 raft_consensus.cc:2454] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P db1a0000aaa346f79c0aa4b6c5105191 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 2.
I20240503 02:00:22.229274 7210 leader_election.cc:302] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46, db1a0000aaa346f79c0aa4b6c5105191; no voters: bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:22.229676 22042 raft_consensus.cc:2781] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:22.229808 22042 raft_consensus.cc:686] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a [term 2 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 02:00:22.230350 22042 consensus_queue.cc:237] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:22.235388 6732 catalog_manager.cc:5484] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 1 to 2, leader changed from bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 2 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:22.252501 21176 raft_consensus.cc:482] T 1b6f98b01ead4cea8f362dc283caa901 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.252694 21176 raft_consensus.cc:504] T 1b6f98b01ead4cea8f362dc283caa901 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:22.253414 21176 leader_election.cc:288] T 1b6f98b01ead4cea8f362dc283caa901 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:22.253647 22255 raft_consensus.cc:482] T 9d44274085b9441c9f15676d842a0710 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.253767 22255 raft_consensus.cc:504] T 9d44274085b9441c9f15676d842a0710 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:22.254241 22255 leader_election.cc:288] T 9d44274085b9441c9f15676d842a0710 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
W20240503 02:00:22.254891 6818 leader_election.cc:334] T 1b6f98b01ead4cea8f362dc283caa901 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.255354 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d44274085b9441c9f15676d842a0710" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 02:00:22.255838 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6f98b01ead4cea8f362dc283caa901" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 02:00:22.256582 6817 leader_election.cc:302] T 1b6f98b01ead4cea8f362dc283caa901 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, db1a0000aaa346f79c0aa4b6c5105191
W20240503 02:00:22.256584 9692 leader_election.cc:334] T 9d44274085b9441c9f15676d842a0710 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.256705 9692 leader_election.cc:302] T 9d44274085b9441c9f15676d842a0710 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, 63331f455be049c1bdd6c23c59ac1bef
I20240503 02:00:22.256901 21176 raft_consensus.cc:2726] T 1b6f98b01ead4cea8f362dc283caa901 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.257629 22255 raft_consensus.cc:2726] T 9d44274085b9441c9f15676d842a0710 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.264742 22255 raft_consensus.cc:482] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.264912 22255 raft_consensus.cc:504] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:22.265278 22255 leader_election.cc:288] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:22.266479 9692 leader_election.cc:334] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:22.266645 9692 leader_election.cc:334] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:22.266686 9692 leader_election.cc:302] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:22.266891 22255 raft_consensus.cc:2726] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.268060 22255 raft_consensus.cc:482] T dccce42f6cd64934808d8c6e5f366e35 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.268172 22255 raft_consensus.cc:504] T dccce42f6cd64934808d8c6e5f366e35 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:22.268537 22255 leader_election.cc:288] T dccce42f6cd64934808d8c6e5f366e35 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:22.269032 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dccce42f6cd64934808d8c6e5f366e35" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
W20240503 02:00:22.270430 9692 leader_election.cc:334] T dccce42f6cd64934808d8c6e5f366e35 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:22.270545 9692 leader_election.cc:302] T dccce42f6cd64934808d8c6e5f366e35 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 5dcc33cc621a4cbbae155f3230fb7c46, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:22.270787 22255 raft_consensus.cc:2726] T dccce42f6cd64934808d8c6e5f366e35 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:22.272938 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:22.293394 22045 raft_consensus.cc:482] T f454ed0b1a924bec9d08f8740137be9b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.293674 22045 raft_consensus.cc:504] T f454ed0b1a924bec9d08f8740137be9b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:22.294163 22045 leader_election.cc:288] T f454ed0b1a924bec9d08f8740137be9b P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 02:00:22.294636 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f454ed0b1a924bec9d08f8740137be9b" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
W20240503 02:00:22.296586 7211 leader_election.cc:334] T f454ed0b1a924bec9d08f8740137be9b P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.296679 7211 leader_election.cc:302] T f454ed0b1a924bec9d08f8740137be9b P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a; no voters: 12e8c683db054df3b75c6285b98a5ad8, 5dcc33cc621a4cbbae155f3230fb7c46
I20240503 02:00:22.296921 22045 raft_consensus.cc:2726] T f454ed0b1a924bec9d08f8740137be9b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.307317 22255 raft_consensus.cc:482] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:22.307524 22255 raft_consensus.cc:504] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:22.308178 22255 leader_election.cc:288] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 02:00:22.308754 22255 raft_consensus.cc:482] T 02e8cbe02d1b4017909e9bfddaf093d8 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.308733 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1df18d990ee43a596a4656dd498b8e7" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 02:00:22.308840 22255 raft_consensus.cc:504] T 02e8cbe02d1b4017909e9bfddaf093d8 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:22.308907 7143 raft_consensus.cc:2454] T d1df18d990ee43a596a4656dd498b8e7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 02:00:22.309288 22255 leader_election.cc:288] T 02e8cbe02d1b4017909e9bfddaf093d8 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 02:00:22.309768 9692 leader_election.cc:302] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 02:00:22.309950 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02e8cbe02d1b4017909e9bfddaf093d8" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 02:00:22.311942 22045 raft_consensus.cc:482] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.311837 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02e8cbe02d1b4017909e9bfddaf093d8" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 02:00:22.312101 7012 raft_consensus.cc:2454] T 02e8cbe02d1b4017909e9bfddaf093d8 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 02:00:22.312077 22045 raft_consensus.cc:504] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:22.312594 22045 leader_election.cc:288] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
W20240503 02:00:22.312762 9690 leader_election.cc:334] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:22.313210 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02e8cbe02d1b4017909e9bfddaf093d8" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 02:00:22.313345 9756 raft_consensus.cc:2454] T 02e8cbe02d1b4017909e9bfddaf093d8 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 02:00:22.313629 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02e8cbe02d1b4017909e9bfddaf093d8" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 02:00:22.313758 7012 raft_consensus.cc:2454] T 02e8cbe02d1b4017909e9bfddaf093d8 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 02:00:22.314149 7209 leader_election.cc:302] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 63331f455be049c1bdd6c23c59ac1bef, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 02:00:22.314697 22042 raft_consensus.cc:2781] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:00:22.314765 22042 raft_consensus.cc:482] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.314803 22042 raft_consensus.cc:3037] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:00:22.314860 9692 leader_election.cc:334] T 02e8cbe02d1b4017909e9bfddaf093d8 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.314927 9692 leader_election.cc:302] T 02e8cbe02d1b4017909e9bfddaf093d8 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, 1d0e738c34af40fa9950c6f086619b3a
I20240503 02:00:22.315295 22233 raft_consensus.cc:2781] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:00:22.315385 22233 raft_consensus.cc:482] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:22.315459 22233 raft_consensus.cc:3037] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.316084 22042 raft_consensus.cc:504] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:22.316241 22255 raft_consensus.cc:2726] T 02e8cbe02d1b4017909e9bfddaf093d8 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.316612 22042 leader_election.cc:288] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 election: Requested vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 02:00:22.317167 22233 raft_consensus.cc:504] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:22.317706 22233 leader_election.cc:288] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 election: Requested vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 02:00:22.318082 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02e8cbe02d1b4017909e9bfddaf093d8" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 02:00:22.318224 9756 raft_consensus.cc:3037] T 02e8cbe02d1b4017909e9bfddaf093d8 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.319411 9756 raft_consensus.cc:2454] T 02e8cbe02d1b4017909e9bfddaf093d8 P db1a0000aaa346f79c0aa4b6c5105191 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 2.
W20240503 02:00:22.321379 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:22.321796 9690 leader_election.cc:334] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:22.321738 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02e8cbe02d1b4017909e9bfddaf093d8" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 02:00:22.321892 7012 raft_consensus.cc:3037] T 02e8cbe02d1b4017909e9bfddaf093d8 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:00:22.322844 7211 leader_election.cc:334] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:22.323035 7211 leader_election.cc:334] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.323184 7012 raft_consensus.cc:2454] T 02e8cbe02d1b4017909e9bfddaf093d8 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 2.
I20240503 02:00:22.323565 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1df18d990ee43a596a4656dd498b8e7" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 02:00:22.323704 7143 raft_consensus.cc:3037] T d1df18d990ee43a596a4656dd498b8e7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.324185 22233 raft_consensus.cc:482] T 65b41c24a2504ad386e437c24f1decee P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.324290 22233 raft_consensus.cc:504] T 65b41c24a2504ad386e437c24f1decee P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:22.324662 22233 leader_election.cc:288] T 65b41c24a2504ad386e437c24f1decee P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:22.324920 7143 raft_consensus.cc:2454] T d1df18d990ee43a596a4656dd498b8e7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 2.
I20240503 02:00:22.325371 9692 leader_election.cc:302] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a, db1a0000aaa346f79c0aa4b6c5105191; no voters: bb5364d730cb4489a0bfc6c270b7450a
W20240503 02:00:22.326483 9692 leader_election.cc:334] T 65b41c24a2504ad386e437c24f1decee P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:22.326684 22233 raft_consensus.cc:2781] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:22.326773 22233 raft_consensus.cc:686] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 [term 2 LEADER]: Becoming Leader. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Running, Role: LEADER
I20240503 02:00:22.326922 22233 consensus_queue.cc:237] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:22.327528 7209 leader_election.cc:302] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 63331f455be049c1bdd6c23c59ac1bef, db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8
I20240503 02:00:22.327955 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b41c24a2504ad386e437c24f1decee" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 02:00:22.328588 22045 raft_consensus.cc:2781] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:22.328684 22045 raft_consensus.cc:686] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a [term 2 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 02:00:22.328817 22045 consensus_queue.cc:237] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:22.328866 9690 leader_election.cc:302] T 65b41c24a2504ad386e437c24f1decee P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 63331f455be049c1bdd6c23c59ac1bef, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:22.329269 22255 raft_consensus.cc:2726] T 65b41c24a2504ad386e437c24f1decee P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.329524 22255 raft_consensus.cc:482] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.329604 22255 raft_consensus.cc:504] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:22.329919 22255 leader_election.cc:288] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 02:00:22.330695 20428 raft_consensus.cc:482] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.330799 20428 raft_consensus.cc:504] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:22.331161 20428 leader_election.cc:288] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:22.331204 9690 leader_election.cc:334] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:22.331696 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f454ed0b1a924bec9d08f8740137be9b" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 02:00:22.331832 7274 raft_consensus.cc:2454] T f454ed0b1a924bec9d08f8740137be9b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 02:00:22.332124 6817 leader_election.cc:302] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 02:00:22.332301 21176 raft_consensus.cc:2781] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:00:22.332347 21176 raft_consensus.cc:482] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.332386 21176 raft_consensus.cc:3037] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.332594 6732 catalog_manager.cc:5484] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: term changed from 1 to 2, leader changed from bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) to db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
W20240503 02:00:22.332828 9692 leader_election.cc:334] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.332888 9692 leader_election.cc:302] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:22.333084 22255 raft_consensus.cc:2726] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.333741 21176 raft_consensus.cc:504] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:22.334116 21176 leader_election.cc:288] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:22.335851 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f454ed0b1a924bec9d08f8740137be9b" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 02:00:22.335983 7274 raft_consensus.cc:3037] T f454ed0b1a924bec9d08f8740137be9b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.337077 7274 raft_consensus.cc:2454] T f454ed0b1a924bec9d08f8740137be9b P 1d0e738c34af40fa9950c6f086619b3a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 2.
I20240503 02:00:22.337435 6817 leader_election.cc:302] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 02:00:22.337621 20428 raft_consensus.cc:2781] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:22.337721 20428 raft_consensus.cc:686] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 [term 2 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 02:00:22.337872 20428 consensus_queue.cc:237] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:22.339413 6740 catalog_manager.cc:5484] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 1 to 2, leader changed from 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 2 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
W20240503 02:00:22.340175 6818 leader_election.cc:334] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.340389 21540 raft_consensus.cc:482] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:22.340489 21540 raft_consensus.cc:504] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
W20240503 02:00:22.341085 6818 leader_election.cc:334] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.341245 21540 leader_election.cc:288] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:22.343713 7080 leader_election.cc:334] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:22.343883 7080 leader_election.cc:334] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.343919 7080 leader_election.cc:302] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:22.344149 21292 raft_consensus.cc:2726] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.344141 22558 raft_consensus.cc:482] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.344244 22558 raft_consensus.cc:504] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:22.344929 22558 leader_election.cc:288] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:22.345292 6735 catalog_manager.cc:5484] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 1 to 2, leader changed from 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:22.345607 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b886572c6ec9473bbec744cee8de12f5" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 02:00:22.345866 7143 raft_consensus.cc:2454] T b886572c6ec9473bbec744cee8de12f5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 02:00:22.346264 22042 raft_consensus.cc:482] T 8db6c1313c0042d48416b4f38fe7166f P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.346395 22042 raft_consensus.cc:504] T 8db6c1313c0042d48416b4f38fe7166f P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:22.347354 22042 leader_election.cc:288] T 8db6c1313c0042d48416b4f38fe7166f P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 02:00:22.347669 6818 leader_election.cc:302] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 02:00:22.347841 21292 raft_consensus.cc:482] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.347901 22558 raft_consensus.cc:2781] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:00:22.347951 22558 raft_consensus.cc:482] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.347995 22558 raft_consensus.cc:3037] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.347959 21292 raft_consensus.cc:504] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:22.348377 21292 leader_election.cc:288] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
W20240503 02:00:22.349053 7078 leader_election.cc:334] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:22.349089 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db6c1313c0042d48416b4f38fe7166f" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 02:00:22.349261 7274 raft_consensus.cc:2454] T 8db6c1313c0042d48416b4f38fe7166f P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 1.
I20240503 02:00:22.349275 22558 raft_consensus.cc:504] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:22.349642 22558 leader_election.cc:288] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 election: Requested vote from peers c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:22.349658 7079 leader_election.cc:302] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 02:00:22.349841 21292 raft_consensus.cc:2781] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:00:22.349913 21292 raft_consensus.cc:482] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting leader election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.349957 21292 raft_consensus.cc:3037] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:00:22.350098 7080 leader_election.cc:334] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:22.350133 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db6c1313c0042d48416b4f38fe7166f" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 02:00:22.350369 22042 raft_consensus.cc:482] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.350461 22042 raft_consensus.cc:504] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:22.350822 22042 leader_election.cc:288] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:22.351123 7211 leader_election.cc:334] T 8db6c1313c0042d48416b4f38fe7166f P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:22.351125 21292 raft_consensus.cc:504] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:22.352160 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db6c1313c0042d48416b4f38fe7166f" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 02:00:22.352303 7274 raft_consensus.cc:3037] T 8db6c1313c0042d48416b4f38fe7166f P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:00:22.352401 7211 leader_election.cc:334] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:22.352495 7211 leader_election.cc:334] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.352542 7211 leader_election.cc:302] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:22.352769 22042 raft_consensus.cc:2726] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:22.353330 6818 leader_election.cc:334] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.354072 7274 raft_consensus.cc:2454] T 8db6c1313c0042d48416b4f38fe7166f P 1d0e738c34af40fa9950c6f086619b3a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in term 2.
W20240503 02:00:22.356336 6818 leader_election.cc:334] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.351531 21292 leader_election.cc:288] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 02:00:22.351908 7143 raft_consensus.cc:2379] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1d0e738c34af40fa9950c6f086619b3a in current term 2: Already voted for candidate c688f3df56ac4b6d9b0c30fa4cc8ca2a in this term.
I20240503 02:00:22.357992 7211 leader_election.cc:302] T 8db6c1313c0042d48416b4f38fe7166f P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, c688f3df56ac4b6d9b0c30fa4cc8ca2a
I20240503 02:00:22.358269 22042 raft_consensus.cc:2726] T 8db6c1313c0042d48416b4f38fe7166f P 1d0e738c34af40fa9950c6f086619b3a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.354748 7079 leader_election.cc:302] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters:
I20240503 02:00:22.360167 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b886572c6ec9473bbec744cee8de12f5" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 02:00:22.360324 7143 raft_consensus.cc:3037] T b886572c6ec9473bbec744cee8de12f5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.361691 7143 raft_consensus.cc:2454] T b886572c6ec9473bbec744cee8de12f5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 2.
I20240503 02:00:22.362075 21292 raft_consensus.cc:2781] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:22.362169 21292 raft_consensus.cc:686] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 2 LEADER]: Becoming Leader. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Running, Role: LEADER
I20240503 02:00:22.362314 21292 consensus_queue.cc:237] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
W20240503 02:00:22.364632 7080 leader_election.cc:334] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:22.365075 6818 leader_election.cc:302] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8
I20240503 02:00:22.365355 22558 raft_consensus.cc:2781] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:22.365852 22558 raft_consensus.cc:686] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 2 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 02:00:22.366075 22558 consensus_queue.cc:237] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:22.366235 6739 catalog_manager.cc:5484] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a reported cstate change: term changed from 1 to 2, leader changed from b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) to c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131). New cstate: current_term: 2 leader_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:22.369933 6735 catalog_manager.cc:5484] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 1 to 2, leader changed from 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:22.373040 22255 raft_consensus.cc:482] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.373255 22255 raft_consensus.cc:504] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:22.374162 22255 leader_election.cc:288] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 02:00:22.377449 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5097893b51da40059b0ef865d42c4d4a" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 02:00:22.377657 6882 raft_consensus.cc:2454] T 5097893b51da40059b0ef865d42c4d4a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 02:00:22.377996 22255 raft_consensus.cc:482] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.378110 22255 raft_consensus.cc:504] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:22.378204 9693 leader_election.cc:302] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 02:00:22.378410 22233 raft_consensus.cc:2781] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:00:22.378463 22233 raft_consensus.cc:482] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.378509 22233 raft_consensus.cc:3037] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:00:22.380564 9692 leader_election.cc:334] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.380427 22233 raft_consensus.cc:504] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
W20240503 02:00:22.381290 9692 leader_election.cc:334] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:22.381295 21540 raft_consensus.cc:482] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.381385 22233 leader_election.cc:288] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 election: Requested vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 02:00:22.381436 21540 raft_consensus.cc:504] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:22.381609 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b93fe374d2a9496a847480b7824a5937" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 02:00:22.381736 7012 raft_consensus.cc:2454] T b93fe374d2a9496a847480b7824a5937 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 02:00:22.381888 21540 leader_election.cc:288] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:22.382094 21540 raft_consensus.cc:482] T 54e3860e5cf54663b4538b9e0f96664e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.382186 21540 raft_consensus.cc:504] T 54e3860e5cf54663b4538b9e0f96664e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:22.382282 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5097893b51da40059b0ef865d42c4d4a" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 02:00:22.382395 6882 raft_consensus.cc:3037] T 5097893b51da40059b0ef865d42c4d4a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.382495 21540 leader_election.cc:288] T 54e3860e5cf54663b4538b9e0f96664e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:22.383098 6883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e3860e5cf54663b4538b9e0f96664e" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
W20240503 02:00:22.382447 9692 leader_election.cc:334] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.383884 6882 raft_consensus.cc:2454] T 5097893b51da40059b0ef865d42c4d4a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 2.
I20240503 02:00:22.384176 9693 leader_election.cc:302] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8
W20240503 02:00:22.384742 7080 leader_election.cc:334] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:22.384824 7080 leader_election.cc:334] T 54e3860e5cf54663b4538b9e0f96664e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.384871 7080 leader_election.cc:302] T 54e3860e5cf54663b4538b9e0f96664e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, 5dcc33cc621a4cbbae155f3230fb7c46
I20240503 02:00:22.384943 9690 leader_election.cc:302] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, db1a0000aaa346f79c0aa4b6c5105191; no voters: b7062fbbc2b54fff9e99ec4ea5562c76
W20240503 02:00:22.385079 7080 leader_election.cc:334] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:22.378465 22255 leader_election.cc:288] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 02:00:22.385133 7080 leader_election.cc:302] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:22.385269 22255 raft_consensus.cc:2781] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:22.385296 21292 raft_consensus.cc:2726] T 54e3860e5cf54663b4538b9e0f96664e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.385350 22255 raft_consensus.cc:686] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 [term 2 LEADER]: Becoming Leader. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Running, Role: LEADER
I20240503 02:00:22.385413 21292 raft_consensus.cc:2726] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.385499 22255 consensus_queue.cc:237] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:22.385704 22562 raft_consensus.cc:2781] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:00:22.385761 22562 raft_consensus.cc:482] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.385833 22562 raft_consensus.cc:3037] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.389202 21721 raft_consensus.cc:482] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:22.389317 21721 raft_consensus.cc:504] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:22.389494 6737 catalog_manager.cc:5484] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: term changed from 1 to 2, leader changed from 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) to db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } }
I20240503 02:00:22.389778 21721 leader_election.cc:288] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:22.389070 22562 raft_consensus.cc:504] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
W20240503 02:00:22.391080 6947 leader_election.cc:334] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:22.391769 22562 leader_election.cc:288] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
W20240503 02:00:22.393966 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:22.394073 22558 raft_consensus.cc:482] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.394158 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b93fe374d2a9496a847480b7824a5937" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 02:00:22.394188 22558 raft_consensus.cc:504] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:22.394296 7014 raft_consensus.cc:3037] T b93fe374d2a9496a847480b7824a5937 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:00:22.394428 9692 leader_election.cc:334] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:22.395385 22558 leader_election.cc:288] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:22.395385 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dccce42f6cd64934808d8c6e5f366e35" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 02:00:22.395525 9756 raft_consensus.cc:2454] T dccce42f6cd64934808d8c6e5f366e35 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
W20240503 02:00:22.396064 6949 leader_election.cc:334] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:22.396158 6949 leader_election.cc:302] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
W20240503 02:00:22.396234 6818 leader_election.cc:334] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:22.396076 7014 raft_consensus.cc:2454] T b93fe374d2a9496a847480b7824a5937 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 2.
I20240503 02:00:22.396399 21721 raft_consensus.cc:2726] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.396433 6817 leader_election.cc:302] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, db1a0000aaa346f79c0aa4b6c5105191; no voters: b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:22.396514 9690 leader_election.cc:302] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, db1a0000aaa346f79c0aa4b6c5105191; no voters: b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:22.396673 22562 raft_consensus.cc:2781] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:22.396762 22562 raft_consensus.cc:686] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 [term 2 LEADER]: Becoming Leader. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Running, Role: LEADER
I20240503 02:00:22.396924 22562 consensus_queue.cc:237] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:22.397748 22558 raft_consensus.cc:2781] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:00:22.397810 22558 raft_consensus.cc:482] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.397855 22558 raft_consensus.cc:3037] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.399281 22558 raft_consensus.cc:504] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:22.400410 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dccce42f6cd64934808d8c6e5f366e35" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 02:00:22.400547 9756 raft_consensus.cc:3037] T dccce42f6cd64934808d8c6e5f366e35 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.401651 9756 raft_consensus.cc:2454] T dccce42f6cd64934808d8c6e5f366e35 P db1a0000aaa346f79c0aa4b6c5105191 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 2.
I20240503 02:00:22.401618 6741 catalog_manager.cc:5484] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: term changed from 1 to 2, leader changed from b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) to db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:22.402484 6817 leader_election.cc:302] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 02:00:22.402716 21176 raft_consensus.cc:2781] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:22.402793 21176 raft_consensus.cc:686] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 2 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 02:00:22.402951 21176 consensus_queue.cc:237] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
W20240503 02:00:22.403944 6818 leader_election.cc:334] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:22.405498 22558 leader_election.cc:288] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 election: Requested vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:22.410691 6740 catalog_manager.cc:5484] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 1 to 2, leader changed from b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:22.414696 21721 raft_consensus.cc:482] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:22.414868 21721 raft_consensus.cc:504] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:22.415350 21721 leader_election.cc:288] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:22.415556 22045 raft_consensus.cc:482] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.415764 22045 raft_consensus.cc:504] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:22.416229 22045 leader_election.cc:288] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:22.417006 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d182ce2c5084082a0990ce8090eb88b" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 02:00:22.417167 9756 raft_consensus.cc:2454] T 1d182ce2c5084082a0990ce8090eb88b P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
W20240503 02:00:22.418046 6947 leader_election.cc:334] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:22.418956 7210 leader_election.cc:302] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 02:00:22.420542 22042 raft_consensus.cc:2781] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:00:22.420634 22042 raft_consensus.cc:482] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.420675 22042 raft_consensus.cc:3037] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.421450 22255 raft_consensus.cc:482] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.421553 22255 raft_consensus.cc:504] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:22.421978 22255 leader_election.cc:288] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 02:00:22.421969 22042 raft_consensus.cc:504] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:22.422313 22042 leader_election.cc:288] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 election: Requested vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:22.422478 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6f98b01ead4cea8f362dc283caa901" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 02:00:22.422653 6882 raft_consensus.cc:2454] T 1b6f98b01ead4cea8f362dc283caa901 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 02:00:22.423020 9693 leader_election.cc:302] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 02:00:22.423089 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d182ce2c5084082a0990ce8090eb88b" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 02:00:22.423198 22255 raft_consensus.cc:2781] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:00:22.423262 22255 raft_consensus.cc:482] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.423302 22255 raft_consensus.cc:3037] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.423210 9756 raft_consensus.cc:3037] T 1d182ce2c5084082a0990ce8090eb88b P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.424607 22255 raft_consensus.cc:504] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:22.424981 22255 leader_election.cc:288] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 election: Requested vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 02:00:22.425885 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6f98b01ead4cea8f362dc283caa901" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 02:00:22.426172 6882 raft_consensus.cc:3037] T 1b6f98b01ead4cea8f362dc283caa901 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.426450 9756 raft_consensus.cc:2454] T 1d182ce2c5084082a0990ce8090eb88b P db1a0000aaa346f79c0aa4b6c5105191 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 2.
I20240503 02:00:22.427269 7210 leader_election.cc:302] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 02:00:22.428174 22045 raft_consensus.cc:2781] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:22.428328 22045 raft_consensus.cc:686] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a [term 2 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 02:00:22.428532 22045 consensus_queue.cc:237] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:22.429118 6882 raft_consensus.cc:2454] T 1b6f98b01ead4cea8f362dc283caa901 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 2.
I20240503 02:00:22.429600 22558 raft_consensus.cc:482] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.429751 22558 raft_consensus.cc:504] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
W20240503 02:00:22.431277 7211 leader_election.cc:334] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:22.431392 6818 leader_election.cc:334] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:22.431474 7211 leader_election.cc:334] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:22.431903 9693 leader_election.cc:302] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 02:00:22.432068 22558 leader_election.cc:288] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 02:00:22.432116 22562 raft_consensus.cc:482] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:22.432209 22562 raft_consensus.cc:504] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:22.432267 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e3860e5cf54663b4538b9e0f96664e" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 02:00:22.432456 7143 raft_consensus.cc:2454] T 54e3860e5cf54663b4538b9e0f96664e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
I20240503 02:00:22.432560 22562 leader_election.cc:288] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 02:00:22.432650 22562 raft_consensus.cc:2781] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:22.432714 22562 raft_consensus.cc:686] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 [term 2 LEADER]: Becoming Leader. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Running, Role: LEADER
I20240503 02:00:22.432868 22562 consensus_queue.cc:237] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:22.432875 6818 leader_election.cc:302] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8
I20240503 02:00:22.433187 22558 raft_consensus.cc:2781] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:00:22.433276 22558 raft_consensus.cc:482] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.433372 22558 raft_consensus.cc:3037] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.433482 21721 raft_consensus.cc:482] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
W20240503 02:00:22.434777 9690 leader_election.cc:334] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:22.433825 21721 raft_consensus.cc:504] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:22.435850 22558 raft_consensus.cc:504] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
W20240503 02:00:22.437162 9692 leader_election.cc:334] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:22.437326 6818 leader_election.cc:334] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:22.437389 9692 leader_election.cc:334] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:22.438174 9692 leader_election.cc:334] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.438251 9692 leader_election.cc:302] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:22.438293 7143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e3860e5cf54663b4538b9e0f96664e" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 02:00:22.438431 7143 raft_consensus.cc:3037] T 54e3860e5cf54663b4538b9e0f96664e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.438467 22233 raft_consensus.cc:2726] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.439182 6741 catalog_manager.cc:5484] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 1 to 2, leader changed from b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 2 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:22.439679 7143 raft_consensus.cc:2454] T 54e3860e5cf54663b4538b9e0f96664e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 2.
I20240503 02:00:22.437608 21721 leader_election.cc:288] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:22.440061 6736 catalog_manager.cc:5484] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: term changed from 1 to 2, leader changed from 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) to db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } }
I20240503 02:00:22.440665 6818 leader_election.cc:302] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8
I20240503 02:00:22.440896 20428 raft_consensus.cc:2781] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:22.440975 20428 raft_consensus.cc:686] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 [term 2 LEADER]: Becoming Leader. State: Replica: 5dcc33cc621a4cbbae155f3230fb7c46, State: Running, Role: LEADER
I20240503 02:00:22.441115 20428 consensus_queue.cc:237] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:22.444931 22045 raft_consensus.cc:482] T 5b386bc757e44ca588e3ba6ca76c92b6 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.445061 22045 raft_consensus.cc:504] T 5b386bc757e44ca588e3ba6ca76c92b6 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:22.445542 22045 leader_election.cc:288] T 5b386bc757e44ca588e3ba6ca76c92b6 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:22.446945 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b386bc757e44ca588e3ba6ca76c92b6" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 02:00:22.447053 22558 leader_election.cc:288] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 election: Requested vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 02:00:22.450134 21540 raft_consensus.cc:482] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.450296 21540 raft_consensus.cc:504] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:22.450840 21540 leader_election.cc:288] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 02:00:22.451834 20428 raft_consensus.cc:482] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.451973 20428 raft_consensus.cc:504] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:22.452486 20428 leader_election.cc:288] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:22.453860 21721 raft_consensus.cc:482] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.454013 21721 raft_consensus.cc:504] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
W20240503 02:00:22.454185 6816 leader_election.cc:334] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:22.454241 7080 leader_election.cc:334] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:22.454392 7078 leader_election.cc:334] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:22.454439 7078 leader_election.cc:302] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:22.454699 20428 raft_consensus.cc:482] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:22.454829 20428 raft_consensus.cc:504] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:22.455255 20428 leader_election.cc:288] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:22.454481 21721 leader_election.cc:288] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:22.456259 21540 raft_consensus.cc:2726] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.457193 22042 raft_consensus.cc:482] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:22.457311 22042 raft_consensus.cc:504] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
W20240503 02:00:22.457443 6816 leader_election.cc:334] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:22.457741 22042 leader_election.cc:288] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:22.458817 7211 leader_election.cc:334] T 5b386bc757e44ca588e3ba6ca76c92b6 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:22.458895 7211 leader_election.cc:302] T 5b386bc757e44ca588e3ba6ca76c92b6 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, db1a0000aaa346f79c0aa4b6c5105191
I20240503 02:00:22.459295 22042 raft_consensus.cc:2726] T 5b386bc757e44ca588e3ba6ca76c92b6 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:22.459719 6818 leader_election.cc:334] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.459800 6818 leader_election.cc:302] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
W20240503 02:00:22.459952 6818 leader_election.cc:334] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.460000 6818 leader_election.cc:302] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:22.460124 21176 raft_consensus.cc:2726] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.460209 21176 raft_consensus.cc:2726] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.460942 22466 raft_consensus.cc:482] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.461046 22466 raft_consensus.cc:504] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:22.461396 22466 leader_election.cc:288] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:22.462133 7209 leader_election.cc:334] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:22.463410 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b41c24a2504ad386e437c24f1decee" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 02:00:22.463678 9756 raft_consensus.cc:2454] T 65b41c24a2504ad386e437c24f1decee P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 02:00:22.465360 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ec2909a6e484f81aad7b39a374e1ef8" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 02:00:22.465561 7014 raft_consensus.cc:2454] T 5ec2909a6e484f81aad7b39a374e1ef8 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
W20240503 02:00:22.465386 6949 leader_election.cc:334] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:22.465698 6949 leader_election.cc:302] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:22.466037 6948 leader_election.cc:302] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 02:00:22.466354 22466 raft_consensus.cc:2726] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.466471 20428 raft_consensus.cc:482] T 9cc9a609452f4f4aad600381968b9227 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.466614 20428 raft_consensus.cc:504] T 9cc9a609452f4f4aad600381968b9227 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:22.467022 22466 raft_consensus.cc:2781] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:00:22.467068 20428 leader_election.cc:288] T 9cc9a609452f4f4aad600381968b9227 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:22.467099 22466 raft_consensus.cc:482] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.467146 22466 raft_consensus.cc:3037] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.467133 7209 leader_election.cc:302] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 63331f455be049c1bdd6c23c59ac1bef; no voters: bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:22.467438 22045 raft_consensus.cc:2781] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:00:22.467495 22045 raft_consensus.cc:482] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:22.467592 22045 raft_consensus.cc:3037] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:00:22.468971 6818 leader_election.cc:334] T 9cc9a609452f4f4aad600381968b9227 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.469137 22466 raft_consensus.cc:504] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:22.469550 22045 raft_consensus.cc:504] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:22.469923 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cc9a609452f4f4aad600381968b9227" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 02:00:22.470346 22466 leader_election.cc:288] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 election: Requested vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:22.470825 6817 leader_election.cc:302] T 9cc9a609452f4f4aad600381968b9227 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, 1d0e738c34af40fa9950c6f086619b3a
I20240503 02:00:22.471189 22558 raft_consensus.cc:2726] T 9cc9a609452f4f4aad600381968b9227 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.471808 22466 raft_consensus.cc:482] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.471918 22466 raft_consensus.cc:504] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:22.472299 22466 leader_election.cc:288] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:22.472254 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b41c24a2504ad386e437c24f1decee" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 02:00:22.472411 9756 raft_consensus.cc:3037] T 65b41c24a2504ad386e437c24f1decee P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.472712 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd0378b2cff42a681361930f487d834" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 02:00:22.472867 9757 raft_consensus.cc:2454] T 6fd0378b2cff42a681361930f487d834 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 02:00:22.473205 6948 leader_election.cc:302] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 02:00:22.473387 22466 raft_consensus.cc:2781] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:00:22.473448 22466 raft_consensus.cc:482] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.473536 22466 raft_consensus.cc:3037] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.474148 9756 raft_consensus.cc:2454] T 65b41c24a2504ad386e437c24f1decee P db1a0000aaa346f79c0aa4b6c5105191 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 2.
I20240503 02:00:22.474685 6948 leader_election.cc:302] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 02:00:22.475018 22466 raft_consensus.cc:504] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:22.475181 21721 raft_consensus.cc:2781] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:22.475364 21721 raft_consensus.cc:686] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef [term 2 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 02:00:22.475390 22045 leader_election.cc:288] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 election: Requested vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 02:00:22.475538 21721 consensus_queue.cc:237] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:22.475710 22466 leader_election.cc:288] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 election: Requested vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:22.475915 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd0378b2cff42a681361930f487d834" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 02:00:22.476056 9757 raft_consensus.cc:3037] T 6fd0378b2cff42a681361930f487d834 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.477305 9757 raft_consensus.cc:2454] T 6fd0378b2cff42a681361930f487d834 P db1a0000aaa346f79c0aa4b6c5105191 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 2.
I20240503 02:00:22.477705 6948 leader_election.cc:302] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 02:00:22.477962 22589 raft_consensus.cc:2781] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:22.478170 22589 raft_consensus.cc:686] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef [term 2 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
W20240503 02:00:22.478659 7209 leader_election.cc:334] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:22.478832 6949 leader_election.cc:334] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:22.478973 6949 leader_election.cc:334] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:22.479009 6949 leader_election.cc:334] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:22.480185 6949 leader_election.cc:334] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.480302 6732 catalog_manager.cc:5484] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 1 to 2, leader changed from b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 2 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
I20240503 02:00:22.480264 6949 leader_election.cc:302] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:22.480863 22466 raft_consensus.cc:2726] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.481554 22589 consensus_queue.cc:237] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
W20240503 02:00:22.482162 6949 leader_election.cc:334] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.482250 6949 leader_election.cc:302] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:22.482385 22466 raft_consensus.cc:2726] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:00:22.483253 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ec2909a6e484f81aad7b39a374e1ef8" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef"
I20240503 02:00:22.483449 7014 raft_consensus.cc:3037] T 5ec2909a6e484f81aad7b39a374e1ef8 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.484927 7014 raft_consensus.cc:2454] T 5ec2909a6e484f81aad7b39a374e1ef8 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 2.
W20240503 02:00:22.485167 6949 leader_election.cc:334] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:22.485376 6949 leader_election.cc:334] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:22.485575 6949 leader_election.cc:334] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.485845 7209 leader_election.cc:302] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 63331f455be049c1bdd6c23c59ac1bef; no voters: bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:22.486615 22042 raft_consensus.cc:2781] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:22.486879 22042 raft_consensus.cc:686] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a [term 2 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 02:00:22.487042 22042 consensus_queue.cc:237] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } }
I20240503 02:00:22.487419 6740 catalog_manager.cc:5484] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 reported cstate change: term changed from 1 to 2, leader changed from 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) to 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129). New cstate: current_term: 2 leader_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
I20240503 02:00:22.491323 22045 raft_consensus.cc:482] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.491467 22045 raft_consensus.cc:504] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:22.491436 6741 catalog_manager.cc:5484] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 1 to 2, leader changed from bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134) to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 2 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } }
I20240503 02:00:22.492292 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cc9a609452f4f4aad600381968b9227" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 02:00:22.492444 6882 raft_consensus.cc:2454] T 9cc9a609452f4f4aad600381968b9227 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 1.
I20240503 02:00:22.492811 7212 leader_election.cc:302] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters:
I20240503 02:00:22.493009 22042 raft_consensus.cc:2781] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:00:22.493062 22042 raft_consensus.cc:482] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.493104 22042 raft_consensus.cc:3037] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.491441 6732 catalog_manager.cc:5484] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 1 to 2, leader changed from 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 2 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:22.494431 22042 raft_consensus.cc:504] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:22.494869 22042 leader_election.cc:288] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 election: Requested vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:22.495270 22255 raft_consensus.cc:482] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.495393 22255 raft_consensus.cc:504] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:22.495806 22255 leader_election.cc:288] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:22.495879 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cc9a609452f4f4aad600381968b9227" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46"
I20240503 02:00:22.496001 6882 raft_consensus.cc:3037] T 9cc9a609452f4f4aad600381968b9227 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:00:22.496181 7211 leader_election.cc:334] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.497283 6882 raft_consensus.cc:2454] T 9cc9a609452f4f4aad600381968b9227 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0e738c34af40fa9950c6f086619b3a in term 2.
I20240503 02:00:22.498194 7212 leader_election.cc:302] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8
W20240503 02:00:22.498749 9692 leader_election.cc:334] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:22.498862 9692 leader_election.cc:334] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:22.498903 9692 leader_election.cc:302] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:22.499128 22233 raft_consensus.cc:2726] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.499998 22045 leader_election.cc:288] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:22.500391 22045 raft_consensus.cc:2781] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:22.500461 22045 raft_consensus.cc:686] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a [term 2 LEADER]: Becoming Leader. State: Replica: 1d0e738c34af40fa9950c6f086619b3a, State: Running, Role: LEADER
I20240503 02:00:22.500593 22045 consensus_queue.cc:237] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
W20240503 02:00:22.502751 7211 leader_election.cc:334] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.508109 6738 catalog_manager.cc:5484] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a reported cstate change: term changed from 1 to 2, leader changed from 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) to 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132). New cstate: current_term: 2 leader_uuid: "1d0e738c34af40fa9950c6f086619b3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } }
I20240503 02:00:22.509946 22255 raft_consensus.cc:482] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.510121 22255 raft_consensus.cc:504] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:22.510666 22255 leader_election.cc:288] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:22.512199 22558 raft_consensus.cc:482] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.512336 22558 raft_consensus.cc:504] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:22.512825 22558 leader_election.cc:288] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:22.515003 6818 leader_election.cc:334] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:22.515187 6818 leader_election.cc:334] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.515298 6818 leader_election.cc:302] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:22.515607 22558 raft_consensus.cc:2726] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.517175 21540 raft_consensus.cc:482] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.517293 21540 raft_consensus.cc:504] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:22.517799 21540 leader_election.cc:288] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:22.519219 7080 leader_election.cc:334] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:22.519697 9692 leader_election.cc:334] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:22.520630 7078 leader_election.cc:334] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:22.519889 9692 leader_election.cc:334] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.520727 7078 leader_election.cc:302] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:22.520771 9692 leader_election.cc:302] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:22.520890 21540 raft_consensus.cc:2726] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.521085 22255 raft_consensus.cc:2726] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.524652 22592 raft_consensus.cc:482] T aa71e80f211f4cce97188f6ac8fd3e8c P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.524784 22592 raft_consensus.cc:504] T aa71e80f211f4cce97188f6ac8fd3e8c P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:22.525235 22592 leader_election.cc:288] T aa71e80f211f4cce97188f6ac8fd3e8c P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 02:00:22.526189 7014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa71e80f211f4cce97188f6ac8fd3e8c" candidate_uuid: "1d0e738c34af40fa9950c6f086619b3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
W20240503 02:00:22.532215 7211 leader_election.cc:334] T aa71e80f211f4cce97188f6ac8fd3e8c P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:22.532399 7211 leader_election.cc:302] T aa71e80f211f4cce97188f6ac8fd3e8c P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a; no voters: 63331f455be049c1bdd6c23c59ac1bef, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:22.533130 22592 raft_consensus.cc:2726] T aa71e80f211f4cce97188f6ac8fd3e8c P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.540443 7254 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 02:00:22.541105 22466 raft_consensus.cc:482] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.541216 22466 raft_consensus.cc:504] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:22.541237 21721 raft_consensus.cc:482] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.541407 21721 raft_consensus.cc:504] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:22.541610 22466 leader_election.cc:288] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:22.541883 21721 leader_election.cc:288] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 02:00:22.541941 21540 raft_consensus.cc:482] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:22.542070 21540 raft_consensus.cc:504] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:22.542354 6992 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 02:00:22.542419 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d44274085b9441c9f15676d842a0710" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 02:00:22.542572 21540 leader_election.cc:288] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:22.542636 9756 raft_consensus.cc:2454] T 9d44274085b9441c9f15676d842a0710 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 02:00:22.543287 6948 leader_election.cc:302] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, db1a0000aaa346f79c0aa4b6c5105191; no voters:
I20240503 02:00:22.543766 21721 raft_consensus.cc:2781] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:00:22.543841 21721 raft_consensus.cc:482] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:22.543884 21721 raft_consensus.cc:3037] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:00:22.544154 6949 leader_election.cc:334] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:22.544245 6949 leader_election.cc:334] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.544277 6949 leader_election.cc:302] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
W20240503 02:00:22.544423 6949 leader_election.cc:334] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.545198 21721 raft_consensus.cc:504] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:22.545612 21721 leader_election.cc:288] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 election: Requested vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053)
I20240503 02:00:22.545711 21721 raft_consensus.cc:2726] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:22.547466 6949 leader_election.cc:334] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:22.547942 7080 leader_election.cc:334] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.549091 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d44274085b9441c9f15676d842a0710" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191"
I20240503 02:00:22.549300 9756 raft_consensus.cc:3037] T 9d44274085b9441c9f15676d842a0710 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:00:22.549522 7078 leader_election.cc:334] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:22.549597 7078 leader_election.cc:302] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:22.549865 21540 raft_consensus.cc:2726] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.550978 9756 raft_consensus.cc:2454] T 9d44274085b9441c9f15676d842a0710 P db1a0000aaa346f79c0aa4b6c5105191 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 2.
I20240503 02:00:22.553822 6861 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 02:00:22.554152 6948 leader_election.cc:302] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8
I20240503 02:00:22.555454 22466 raft_consensus.cc:2781] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:22.555567 22466 raft_consensus.cc:686] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef [term 2 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 02:00:22.555732 22466 consensus_queue.cc:237] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:22.564085 6740 catalog_manager.cc:5484] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 1 to 2, leader changed from 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133) to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 2 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:22.571369 7123 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:22.584687 21540 raft_consensus.cc:482] T d4fa0337228d4f8cb6b6c3aab437619e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.584836 21540 raft_consensus.cc:504] T d4fa0337228d4f8cb6b6c3aab437619e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:22.585268 21540 leader_election.cc:288] T d4fa0337228d4f8cb6b6c3aab437619e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:22.586225 9735 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 02:00:22.587257 7080 leader_election.cc:334] T d4fa0337228d4f8cb6b6c3aab437619e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:22.587554 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fa0337228d4f8cb6b6c3aab437619e" candidate_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "63331f455be049c1bdd6c23c59ac1bef" is_pre_election: true
I20240503 02:00:22.588049 7078 leader_election.cc:302] T d4fa0337228d4f8cb6b6c3aab437619e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 63331f455be049c1bdd6c23c59ac1bef, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:22.588317 21540 raft_consensus.cc:2726] T d4fa0337228d4f8cb6b6c3aab437619e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.604146 22233 raft_consensus.cc:482] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.604282 22233 raft_consensus.cc:504] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:22.604750 22233 leader_election.cc:288] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:22.605758 9692 leader_election.cc:334] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:22.606016 9690 leader_election.cc:334] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:22.606086 9690 leader_election.cc:302] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:22.606370 22233 raft_consensus.cc:2726] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.631541 22233 raft_consensus.cc:482] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.631678 22233 raft_consensus.cc:504] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:22.632120 22233 leader_election.cc:288] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:22.632748 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b386bc757e44ca588e3ba6ca76c92b6" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 02:00:22.632997 7274 raft_consensus.cc:2454] T 5b386bc757e44ca588e3ba6ca76c92b6 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 02:00:22.633430 9691 leader_election.cc:302] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, db1a0000aaa346f79c0aa4b6c5105191; no voters:
W20240503 02:00:22.633694 9692 leader_election.cc:334] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:22.633914 22233 raft_consensus.cc:2781] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:00:22.633965 22233 raft_consensus.cc:482] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.634012 22233 raft_consensus.cc:3037] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.635217 22233 raft_consensus.cc:504] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:22.635608 22233 leader_election.cc:288] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 election: Requested vote from peers 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:22.636030 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b386bc757e44ca588e3ba6ca76c92b6" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 02:00:22.636196 7274 raft_consensus.cc:3037] T 5b386bc757e44ca588e3ba6ca76c92b6 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:00:22.637071 9692 leader_election.cc:334] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:22.637394 7274 raft_consensus.cc:2454] T 5b386bc757e44ca588e3ba6ca76c92b6 P 1d0e738c34af40fa9950c6f086619b3a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 2.
I20240503 02:00:22.637902 9691 leader_election.cc:302] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, db1a0000aaa346f79c0aa4b6c5105191; no voters: b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:22.638173 22233 raft_consensus.cc:2781] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:22.638315 22233 raft_consensus.cc:686] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 [term 2 LEADER]: Becoming Leader. State: Replica: db1a0000aaa346f79c0aa4b6c5105191, State: Running, Role: LEADER
I20240503 02:00:22.638476 22233 consensus_queue.cc:237] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:22.640714 22466 raft_consensus.cc:482] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.640872 22466 raft_consensus.cc:504] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:22.641300 22466 leader_election.cc:288] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 02:00:22.641793 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa71e80f211f4cce97188f6ac8fd3e8c" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a" is_pre_election: true
I20240503 02:00:22.641939 7274 raft_consensus.cc:2454] T aa71e80f211f4cce97188f6ac8fd3e8c P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
I20240503 02:00:22.642284 6948 leader_election.cc:302] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 63331f455be049c1bdd6c23c59ac1bef; no voters:
I20240503 02:00:22.642457 22466 raft_consensus.cc:2781] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:00:22.642561 22466 raft_consensus.cc:482] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.642616 22466 raft_consensus.cc:3037] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.643154 6740 catalog_manager.cc:5484] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 reported cstate change: term changed from 1 to 2, leader changed from b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) to db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136). New cstate: current_term: 2 leader_uuid: "db1a0000aaa346f79c0aa4b6c5105191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } }
W20240503 02:00:22.643739 6949 leader_election.cc:334] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:22.643909 22466 raft_consensus.cc:504] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:22.644273 22466 leader_election.cc:288] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 1d0e738c34af40fa9950c6f086619b3a (127.0.99.132:36271)
I20240503 02:00:22.644970 6883 raft_consensus.cc:1261] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 5dcc33cc621a4cbbae155f3230fb7c46 [term 2 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:00:22.646484 22592 consensus_queue.cc:1035] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 02:00:22.646847 6949 leader_election.cc:334] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:22.648964 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa71e80f211f4cce97188f6ac8fd3e8c" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1d0e738c34af40fa9950c6f086619b3a"
I20240503 02:00:22.649106 7274 raft_consensus.cc:3037] T aa71e80f211f4cce97188f6ac8fd3e8c P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.650372 7274 raft_consensus.cc:2454] T aa71e80f211f4cce97188f6ac8fd3e8c P 1d0e738c34af40fa9950c6f086619b3a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 2.
I20240503 02:00:22.650735 6948 leader_election.cc:302] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a, 63331f455be049c1bdd6c23c59ac1bef; no voters: b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:22.651005 22466 raft_consensus.cc:2781] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:22.651090 22466 raft_consensus.cc:686] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef [term 2 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 02:00:22.651237 22466 consensus_queue.cc:237] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:22.654999 6740 catalog_manager.cc:5484] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 1 to 2, leader changed from b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 2 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } }
I20240503 02:00:22.657568 22466 raft_consensus.cc:482] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.657729 22466 raft_consensus.cc:504] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:22.658200 22466 leader_election.cc:288] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 02:00:22.659144 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fa0337228d4f8cb6b6c3aab437619e" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" is_pre_election: true
I20240503 02:00:22.659308 7145 raft_consensus.cc:2454] T d4fa0337228d4f8cb6b6c3aab437619e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 1.
W20240503 02:00:22.659446 6949 leader_election.cc:334] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:22.659646 6949 leader_election.cc:302] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:22.659866 22466 raft_consensus.cc:2781] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:00:22.659922 22466 raft_consensus.cc:482] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting leader election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.659979 22466 raft_consensus.cc:3037] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:00:22.660897 22466 raft_consensus.cc:504] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:22.661227 22466 leader_election.cc:288] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 election: Requested vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979)
I20240503 02:00:22.661908 7145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fa0337228d4f8cb6b6c3aab437619e" candidate_uuid: "63331f455be049c1bdd6c23c59ac1bef" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a"
I20240503 02:00:22.662055 7145 raft_consensus.cc:3037] T d4fa0337228d4f8cb6b6c3aab437619e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:00:22.662571 6949 leader_election.cc:334] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:22.663322 7145 raft_consensus.cc:2454] T d4fa0337228d4f8cb6b6c3aab437619e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63331f455be049c1bdd6c23c59ac1bef in term 2.
I20240503 02:00:22.663669 6949 leader_election.cc:302] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef, c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:22.663915 22466 raft_consensus.cc:2781] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:00:22.664284 22466 raft_consensus.cc:686] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef [term 2 LEADER]: Becoming Leader. State: Replica: 63331f455be049c1bdd6c23c59ac1bef, State: Running, Role: LEADER
I20240503 02:00:22.664573 22466 consensus_queue.cc:237] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:22.668067 22558 raft_consensus.cc:482] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:22.668246 22558 raft_consensus.cc:504] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:22.668711 22558 leader_election.cc:288] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:22.668581 6740 catalog_manager.cc:5484] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef reported cstate change: term changed from 1 to 2, leader changed from b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135) to 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130). New cstate: current_term: 2 leader_uuid: "63331f455be049c1bdd6c23c59ac1bef" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } }
W20240503 02:00:22.670452 6818 leader_election.cc:334] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:22.670799 6816 leader_election.cc:334] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:22.670889 6816 leader_election.cc:302] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:22.671242 22558 raft_consensus.cc:2726] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:22.686810 9756 raft_consensus.cc:1261] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P db1a0000aaa346f79c0aa4b6c5105191 [term 2 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:00:22.687307 22592 consensus_queue.cc:1035] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 02:00:22.694144 7209 consensus_peers.cc:485] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:22.743314 7274 raft_consensus.cc:1261] T f454ed0b1a924bec9d08f8740137be9b P 1d0e738c34af40fa9950c6f086619b3a [term 2 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240503 02:00:22.743711 9690 consensus_peers.cc:485] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:22.744287 22558 consensus_queue.cc:1035] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:00:22.750317 6818 consensus_peers.cc:485] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:22.751897 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:22.791694 7012 raft_consensus.cc:1261] T 02e8cbe02d1b4017909e9bfddaf093d8 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:00:22.792158 22042 consensus_queue.cc:1035] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 02:00:22.803872 9692 consensus_peers.cc:485] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:22.810315 9756 raft_consensus.cc:1261] T 1d182ce2c5084082a0990ce8090eb88b P db1a0000aaa346f79c0aa4b6c5105191 [term 2 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:00:22.811054 22042 consensus_queue.cc:1035] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:22.818189 6861 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 02:00:22.822649 7254 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 02:00:22.832053 7211 consensus_peers.cc:485] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:22.847157 6992 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240503 02:00:22.849346 7123 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:22.850831 7145 raft_consensus.cc:1261] T b886572c6ec9473bbec744cee8de12f5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 2 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:00:22.851845 22558 consensus_queue.cc:1035] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:22.854064 9735 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:22.857098 6883 raft_consensus.cc:1261] T 1b6f98b01ead4cea8f362dc283caa901 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 2 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:00:22.857656 22233 consensus_queue.cc:1035] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:00:22.864228 9756 raft_consensus.cc:1261] T 02e8cbe02d1b4017909e9bfddaf093d8 P db1a0000aaa346f79c0aa4b6c5105191 [term 2 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:00:22.864691 22045 consensus_queue.cc:1035] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:00:22.864732 9757 raft_consensus.cc:1261] T dccce42f6cd64934808d8c6e5f366e35 P db1a0000aaa346f79c0aa4b6c5105191 [term 2 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:00:22.865204 22558 consensus_queue.cc:1035] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:00:22.869067 6818 consensus_peers.cc:485] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:22.872964 7211 consensus_peers.cc:485] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:22.874450 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:22.876451 6818 consensus_peers.cc:485] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:22.878917 9692 consensus_peers.cc:485] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:22.879544 7274 raft_consensus.cc:1261] T 8db6c1313c0042d48416b4f38fe7166f P 1d0e738c34af40fa9950c6f086619b3a [term 2 FOLLOWER]: Refusing update from remote peer c688f3df56ac4b6d9b0c30fa4cc8ca2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:00:22.879947 21292 consensus_queue.cc:1035] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:22.886770 6883 raft_consensus.cc:1261] T 9cc9a609452f4f4aad600381968b9227 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 2 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240503 02:00:22.887939 7080 consensus_peers.cc:485] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:22.888016 22042 consensus_queue.cc:1035] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:22.897006 7145 raft_consensus.cc:1261] T d1df18d990ee43a596a4656dd498b8e7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 2 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:00:22.897503 22562 consensus_queue.cc:1035] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:00:22.902370 7211 consensus_peers.cc:485] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:22.904374 6818 consensus_peers.cc:485] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:22.904537 9756 raft_consensus.cc:1261] T 65b41c24a2504ad386e437c24f1decee P db1a0000aaa346f79c0aa4b6c5105191 [term 2 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:00:22.906388 22589 consensus_queue.cc:1035] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:00:22.918062 6949 consensus_peers.cc:485] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:22.938802 9692 consensus_peers.cc:485] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:22.992090 7012 raft_consensus.cc:1261] T b93fe374d2a9496a847480b7824a5937 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:00:22.992626 22233 consensus_queue.cc:1035] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:00:22.994912 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:22.997936 6883 raft_consensus.cc:1261] T 5097893b51da40059b0ef865d42c4d4a P 5dcc33cc621a4cbbae155f3230fb7c46 [term 2 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:00:23.000517 22233 consensus_queue.cc:1035] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:23.027379 9756 raft_consensus.cc:1261] T 9d44274085b9441c9f15676d842a0710 P db1a0000aaa346f79c0aa4b6c5105191 [term 2 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:00:23.028013 22466 consensus_queue.cc:1035] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:23.036659 9757 raft_consensus.cc:1261] T 6fd0378b2cff42a681361930f487d834 P db1a0000aaa346f79c0aa4b6c5105191 [term 2 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:00:23.037112 21721 consensus_queue.cc:1035] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:00:23.042173 6949 consensus_peers.cc:485] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:23.043236 6949 consensus_peers.cc:485] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:23.043753 7012 raft_consensus.cc:1261] T 5ec2909a6e484f81aad7b39a374e1ef8 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Refusing update from remote peer 1d0e738c34af40fa9950c6f086619b3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:00:23.044270 22045 consensus_queue.cc:1035] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:00:23.047122 7143 raft_consensus.cc:1261] T 54e3860e5cf54663b4538b9e0f96664e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 2 FOLLOWER]: Refusing update from remote peer 5dcc33cc621a4cbbae155f3230fb7c46: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:00:23.047652 22558 consensus_queue.cc:1035] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:00:23.051954 7209 consensus_peers.cc:485] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:23.090991 6949 consensus_peers.cc:485] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:23.091924 7272 raft_consensus.cc:1261] T aa71e80f211f4cce97188f6ac8fd3e8c P 1d0e738c34af40fa9950c6f086619b3a [term 2 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:00:23.092566 22589 consensus_queue.cc:1035] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:00:23.096966 6949 consensus_peers.cc:485] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:23.117233 7254 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 02:00:23.123658 7123 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:23.133502 6992 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240503 02:00:23.138545 6861 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 02:00:23.141606 9735 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:23.160403 7272 raft_consensus.cc:1261] T 5b386bc757e44ca588e3ba6ca76c92b6 P 1d0e738c34af40fa9950c6f086619b3a [term 2 FOLLOWER]: Refusing update from remote peer db1a0000aaa346f79c0aa4b6c5105191: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:00:23.161224 22233 consensus_queue.cc:1035] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:00:23.176926 9692 consensus_peers.cc:485] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:23.199628 7143 raft_consensus.cc:1261] T d4fa0337228d4f8cb6b6c3aab437619e P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 2 FOLLOWER]: Refusing update from remote peer 63331f455be049c1bdd6c23c59ac1bef: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:00:23.200100 22589 consensus_queue.cc:1035] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:00:23.207118 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:23.318094 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:23.378233 6816 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:23.398586 7254 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 02:00:23.403182 9735 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:23.403970 6992 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240503 02:00:23.405354 6861 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 02:00:23.408672 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:23.408715 7123 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 02:00:23.457808 7211 consensus_peers.cc:485] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.465363 9692 consensus_peers.cc:485] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.519203 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.574759 9690 consensus_peers.cc:485] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.596262 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.627221 6947 consensus_peers.cc:485] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.628563 7078 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.631466 6949 consensus_peers.cc:485] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:23.650799 7254 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 02:00:23.651373 6861 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 02:00:23.654341 7123 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 02:00:23.663982 7080 consensus_peers.cc:485] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.668438 6947 consensus_peers.cc:485] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:23.669468 6992 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240503 02:00:23.674888 9735 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:00:23.682176 7211 consensus_peers.cc:485] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.682536 6816 consensus_peers.cc:485] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.685307 7080 consensus_peers.cc:485] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.713550 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:23.726081 6818 consensus_peers.cc:485] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.729548 7078 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.741282 7209 consensus_peers.cc:485] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.768733 7078 consensus_peers.cc:485] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.779158 6818 consensus_peers.cc:485] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.784896 6816 consensus_peers.cc:485] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.785634 7209 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.785813 9692 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.787518 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.791108 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.791235 6949 consensus_peers.cc:485] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.797188 6818 consensus_peers.cc:485] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.798380 6818 consensus_peers.cc:485] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.811765 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:23.815819 22562 raft_consensus.cc:482] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:23.816169 22562 raft_consensus.cc:504] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:23.817179 22562 leader_election.cc:288] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 02:00:23.819124 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49eabbb8972840759ffa83951cb0b3b7" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
W20240503 02:00:23.819309 9692 leader_election.cc:334] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:23.819355 6882 raft_consensus.cc:2454] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
I20240503 02:00:23.819980 21292 raft_consensus.cc:482] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
W20240503 02:00:23.820134 9690 leader_election.cc:334] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:23.820195 9690 leader_election.cc:302] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:23.820143 21292 raft_consensus.cc:504] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:23.820390 22562 raft_consensus.cc:2726] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:23.820792 21292 leader_election.cc:288] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:23.828441 7080 leader_election.cc:334] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:23.830648 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.832262 7080 leader_election.cc:334] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:23.833402 7080 leader_election.cc:302] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
W20240503 02:00:23.833554 9692 consensus_peers.cc:485] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:23.833817 21292 raft_consensus.cc:2726] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:23.840224 6949 consensus_peers.cc:485] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.841675 9690 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.866250 7209 consensus_peers.cc:485] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.866672 6818 consensus_peers.cc:485] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.869192 7080 consensus_peers.cc:485] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.869414 7080 consensus_peers.cc:485] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.872870 7080 consensus_peers.cc:485] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:23.880842 22592 raft_consensus.cc:482] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:23.881013 22592 raft_consensus.cc:504] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:23.881847 22592 leader_election.cc:288] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:23.884080 7211 leader_election.cc:334] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:23.887933 7209 leader_election.cc:334] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:23.888068 7209 leader_election.cc:302] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:23.888238 22042 raft_consensus.cc:2726] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:23.892937 7209 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.893072 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:23.917874 6818 consensus_peers.cc:485] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.929481 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.930373 6947 consensus_peers.cc:485] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.940253 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:23.946442 6861 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 02:00:23.948971 7254 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 02:00:23.950891 6992 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
W20240503 02:00:23.968024 7078 consensus_peers.cc:485] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:23.970890 9735 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 02:00:23.971637 7078 consensus_peers.cc:485] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.974839 7211 consensus_peers.cc:485] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:23.975425 7123 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 02:00:23.977348 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.977746 6816 consensus_peers.cc:485] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.977838 7080 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:23.984750 21540 raft_consensus.cc:482] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:23.984925 21540 raft_consensus.cc:504] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:23.985369 21540 leader_election.cc:288] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:23.988185 6818 consensus_peers.cc:485] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.989809 7080 leader_election.cc:334] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:23.990316 7080 leader_election.cc:334] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:23.990454 7080 leader_election.cc:302] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:23.991202 21540 raft_consensus.cc:2726] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:23.992331 6818 consensus_peers.cc:485] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.992597 9692 consensus_peers.cc:485] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.995595 6947 consensus_peers.cc:485] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.995859 7211 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:23.996775 6947 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:24.010186 22466 raft_consensus.cc:482] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:24.010334 22466 raft_consensus.cc:504] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:24.010766 22466 leader_election.cc:288] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:24.012818 6816 consensus_peers.cc:485] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:24.015944 9690 consensus_peers.cc:485] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:24.016160 9692 consensus_peers.cc:485] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:24.016141 7078 consensus_peers.cc:485] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:24.018082 6949 leader_election.cc:334] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:24.018594 6949 leader_election.cc:334] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:24.018657 6949 leader_election.cc:302] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:24.018827 22466 raft_consensus.cc:2726] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:24.026023 7211 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:24.041891 6947 consensus_peers.cc:485] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:24.047513 6949 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:24.047645 6947 consensus_peers.cc:485] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:24.051715 7078 consensus_peers.cc:485] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:24.052726 7211 consensus_peers.cc:485] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:24.053400 6949 consensus_peers.cc:485] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:24.056335 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:24.058283 6818 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:24.061968 20428 raft_consensus.cc:482] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:24.062115 20428 raft_consensus.cc:504] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:24.062598 20428 leader_election.cc:288] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:24.063969 6816 leader_election.cc:334] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:24.064924 6818 leader_election.cc:334] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:24.065047 6818 leader_election.cc:302] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:24.066447 21176 raft_consensus.cc:2726] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:24.072885 9692 consensus_peers.cc:485] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:24.073005 9692 consensus_peers.cc:485] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:24.073241 22914 raft_consensus.cc:482] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:24.073386 22914 raft_consensus.cc:504] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:24.073834 22914 leader_election.cc:288] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:24.076323 6949 leader_election.cc:334] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:24.076526 6949 leader_election.cc:334] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:24.076719 6949 leader_election.cc:302] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:24.076933 22914 raft_consensus.cc:2726] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:24.079439 7080 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:24.084892 22913 raft_consensus.cc:482] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:24.085036 22913 raft_consensus.cc:504] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:24.085767 22913 leader_election.cc:288] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:24.088559 6816 leader_election.cc:334] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:24.088821 6818 leader_election.cc:334] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:24.088946 6818 leader_election.cc:302] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:24.089313 21176 raft_consensus.cc:2726] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:24.090432 6818 consensus_peers.cc:485] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:24.092742 7209 consensus_peers.cc:485] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:24.106544 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:24.111383 6818 consensus_peers.cc:485] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:24.130183 6818 consensus_peers.cc:485] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:24.138830 22914 raft_consensus.cc:482] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:24.138972 22914 raft_consensus.cc:504] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:24.139384 22914 leader_election.cc:288] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:24.140677 21540 raft_consensus.cc:482] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:24.140815 21540 raft_consensus.cc:504] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:24.141301 21540 leader_election.cc:288] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:24.141695 6947 leader_election.cc:334] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:24.142323 6949 leader_election.cc:334] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:24.142756 6949 leader_election.cc:302] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
W20240503 02:00:24.142915 7078 leader_election.cc:334] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:24.143019 22914 raft_consensus.cc:2726] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:24.143064 21540 raft_consensus.cc:482] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:24.143165 21540 raft_consensus.cc:504] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:24.143617 21540 leader_election.cc:288] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:24.150420 7080 leader_election.cc:334] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:24.150581 7080 leader_election.cc:302] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:24.151405 21292 raft_consensus.cc:2726] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:24.152069 7078 leader_election.cc:334] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:24.153338 7080 leader_election.cc:334] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:24.153436 7080 leader_election.cc:302] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:24.153618 21540 raft_consensus.cc:2726] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:24.154062 22255 raft_consensus.cc:482] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:24.154181 22255 raft_consensus.cc:504] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
W20240503 02:00:24.154392 7078 consensus_peers.cc:485] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:24.154677 22255 leader_election.cc:288] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:24.156491 9692 leader_election.cc:334] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:24.156720 9690 leader_election.cc:334] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:24.156965 9690 leader_election.cc:302] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:24.157166 22255 raft_consensus.cc:2726] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:24.180191 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:24.181345 7078 consensus_peers.cc:485] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:24.182911 21176 raft_consensus.cc:482] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:24.183039 21176 raft_consensus.cc:504] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:24.183483 21176 leader_election.cc:288] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:24.185752 6818 leader_election.cc:334] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:24.186136 6816 leader_election.cc:334] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:24.186241 6816 leader_election.cc:302] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:24.186718 21176 raft_consensus.cc:2726] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:24.193576 9690 consensus_peers.cc:485] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:24.210862 7211 consensus_peers.cc:485] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:24.216054 7078 consensus_peers.cc:485] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:24.221261 7211 consensus_peers.cc:485] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:24.222460 22913 raft_consensus.cc:482] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:24.222968 22913 raft_consensus.cc:504] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:24.223714 22913 leader_election.cc:288] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:24.228525 6816 leader_election.cc:334] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:24.228818 6818 leader_election.cc:334] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:24.228902 6818 leader_election.cc:302] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:24.229043 21176 raft_consensus.cc:2726] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:24.236958 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:24.239095 22255 raft_consensus.cc:482] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:24.239252 22255 raft_consensus.cc:504] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:24.239709 22255 leader_election.cc:288] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:24.241946 9692 leader_election.cc:334] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:24.242672 9692 leader_election.cc:334] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:24.242802 9692 leader_election.cc:302] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:24.242959 22562 raft_consensus.cc:2726] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:24.244004 22589 raft_consensus.cc:482] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:24.244110 22589 raft_consensus.cc:504] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:24.244537 22589 leader_election.cc:288] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:24.246834 6947 leader_election.cc:334] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:24.246907 22562 raft_consensus.cc:482] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:24.247107 22562 raft_consensus.cc:504] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:24.247851 22562 leader_election.cc:288] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:24.251066 6949 leader_election.cc:334] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:24.251181 6949 leader_election.cc:302] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:24.251552 22589 raft_consensus.cc:2726] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:24.252763 9692 leader_election.cc:334] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:24.252885 9692 leader_election.cc:334] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:24.252915 9692 leader_election.cc:302] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
W20240503 02:00:24.253033 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:24.253050 22562 raft_consensus.cc:2726] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:24.254587 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:24.273432 21540 raft_consensus.cc:482] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:24.273571 21540 raft_consensus.cc:504] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:24.274029 21540 leader_election.cc:288] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:24.275883 7254 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 02:00:24.277526 6861 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 02:00:24.279220 7078 leader_election.cc:334] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:24.281302 6992 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
W20240503 02:00:24.286618 7080 leader_election.cc:334] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:24.286746 7080 leader_election.cc:302] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
W20240503 02:00:24.286909 7080 leader_election.cc:334] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:24.287155 22945 raft_consensus.cc:2726] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:24.296265 9690 consensus_peers.cc:485] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:24.296346 7123 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:24.303534 9735 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:24.306088 21540 raft_consensus.cc:482] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:24.306216 21540 raft_consensus.cc:504] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:24.306758 21540 leader_election.cc:288] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:24.314388 22589 raft_consensus.cc:482] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:24.314568 22589 raft_consensus.cc:504] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:24.315012 22589 leader_election.cc:288] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:24.316862 6949 leader_election.cc:334] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:24.317027 6949 leader_election.cc:334] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:24.317083 6949 leader_election.cc:302] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:24.317867 22589 raft_consensus.cc:2726] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:24.323774 7080 leader_election.cc:334] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:24.323932 7080 leader_election.cc:334] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:24.323978 7080 leader_election.cc:302] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:24.324453 21540 raft_consensus.cc:2726] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:24.352734 7209 consensus_peers.cc:485] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:24.356568 22255 raft_consensus.cc:482] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:24.356786 22255 raft_consensus.cc:504] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:24.357718 22255 leader_election.cc:288] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:24.358952 9690 leader_election.cc:334] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:24.359422 9692 leader_election.cc:334] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:24.359499 9692 leader_election.cc:302] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:24.359715 22255 raft_consensus.cc:2726] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:24.434900 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:24.462404 22592 raft_consensus.cc:482] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:24.462581 22592 raft_consensus.cc:504] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:24.463016 22592 leader_election.cc:288] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:24.465358 7211 leader_election.cc:334] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:24.466032 7211 leader_election.cc:334] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:24.466141 7211 leader_election.cc:302] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:24.467711 22592 raft_consensus.cc:2726] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:24.509546 22945 raft_consensus.cc:482] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:24.509904 22945 raft_consensus.cc:504] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:24.510452 22945 leader_election.cc:288] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:24.515105 7078 leader_election.cc:334] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:24.516091 7080 leader_election.cc:334] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:24.516219 7080 leader_election.cc:302] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:24.516709 21540 raft_consensus.cc:2726] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:24.521407 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:24.531375 22589 raft_consensus.cc:482] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:24.531515 22589 raft_consensus.cc:504] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:24.531934 22589 leader_election.cc:288] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:24.533233 22255 raft_consensus.cc:482] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:24.533362 22255 raft_consensus.cc:504] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
W20240503 02:00:24.533440 6949 leader_election.cc:334] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:24.533743 6949 leader_election.cc:334] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:24.533804 6949 leader_election.cc:302] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:24.534034 22255 leader_election.cc:288] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
I20240503 02:00:24.534066 22589 raft_consensus.cc:2726] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:00:24.535492 9692 leader_election.cc:334] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:24.535856 9692 leader_election.cc:334] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:24.536020 9692 leader_election.cc:302] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:24.536415 22255 raft_consensus.cc:2726] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:24.578048 22913 raft_consensus.cc:482] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:24.578223 22913 raft_consensus.cc:504] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:24.579123 22913 leader_election.cc:288] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:24.583134 6818 leader_election.cc:334] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:24.583314 6818 leader_election.cc:334] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:24.583359 6818 leader_election.cc:302] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:24.584019 22913 raft_consensus.cc:2726] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:24.587023 7254 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 02:00:24.590077 7123 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 02:00:24.592226 6992 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
I20240503 02:00:24.592698 6861 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 02:00:24.596210 9735 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:00:24.713311 7012 consensus_queue.cc:237] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
W20240503 02:00:24.716459 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:24.716866 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): Couldn't send request to peer c688f3df56ac4b6d9b0c30fa4cc8ca2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 622197c9b8664d94bbae183f6f2fd381. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:00:24.717000 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:00:24.729249 22945 raft_consensus.cc:482] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:24.729506 22945 raft_consensus.cc:504] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:24.730091 22945 leader_election.cc:288] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:24.731451 7078 leader_election.cc:334] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:24.732108 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:24.732232 7080 leader_election.cc:334] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:24.732291 7080 leader_election.cc:302] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:24.732473 22945 raft_consensus.cc:2726] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:24.842424 22562 raft_consensus.cc:482] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:24.842653 22562 raft_consensus.cc:504] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:24.843166 22562 leader_election.cc:288] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:24.845134 9690 leader_election.cc:334] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:24.846048 9692 leader_election.cc:334] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:24.846156 9692 leader_election.cc:302] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:24.846340 22562 raft_consensus.cc:2726] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:24.950212 7209 consensus_peers.cc:485] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:24.953613 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:25.002089 7211 consensus_peers.cc:485] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:25.022961 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:25.187088 6818 consensus_peers.cc:485] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:25.224085 9690 consensus_peers.cc:485] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:25.238569 23064 ts_tablet_manager.cc:902] T 622197c9b8664d94bbae183f6f2fd381 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Initiating tablet copy from peer 63331f455be049c1bdd6c23c59ac1bef (127.0.99.130:37231)
I20240503 02:00:25.239145 23064 tablet_copy_client.cc:308] T 622197c9b8664d94bbae183f6f2fd381 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.130:37231
I20240503 02:00:25.239815 7034 tablet_copy_service.cc:140] P 63331f455be049c1bdd6c23c59ac1bef: Received BeginTabletCopySession request for tablet 622197c9b8664d94bbae183f6f2fd381 from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a ({username='slave'} at 127.0.99.131:60361)
I20240503 02:00:25.239938 7034 tablet_copy_service.cc:161] P 63331f455be049c1bdd6c23c59ac1bef: Beginning new tablet copy session on tablet 622197c9b8664d94bbae183f6f2fd381 from peer c688f3df56ac4b6d9b0c30fa4cc8ca2a at {username='slave'} at 127.0.99.131:60361: session id = c688f3df56ac4b6d9b0c30fa4cc8ca2a-622197c9b8664d94bbae183f6f2fd381
I20240503 02:00:25.241005 7034 tablet_copy_source_session.cc:205] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:00:25.242090 23064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 622197c9b8664d94bbae183f6f2fd381. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:00:25.244621 23064 tablet_copy_client.cc:783] T 622197c9b8664d94bbae183f6f2fd381 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Starting download of 0 data blocks...
I20240503 02:00:25.244890 23064 tablet_copy_client.cc:647] T 622197c9b8664d94bbae183f6f2fd381 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Starting download of 1 WAL segments...
I20240503 02:00:25.246492 23064 tablet_copy_client.cc:523] T 622197c9b8664d94bbae183f6f2fd381 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:00:25.248328 23064 tablet_bootstrap.cc:492] T 622197c9b8664d94bbae183f6f2fd381 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap starting.
I20240503 02:00:25.251946 23064 tablet_bootstrap.cc:492] T 622197c9b8664d94bbae183f6f2fd381 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 02:00:25.252280 23064 tablet_bootstrap.cc:492] T 622197c9b8664d94bbae183f6f2fd381 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Bootstrap complete.
I20240503 02:00:25.252471 23064 ts_tablet_manager.cc:1367] T 622197c9b8664d94bbae183f6f2fd381 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20240503 02:00:25.252936 23064 raft_consensus.cc:348] T 622197c9b8664d94bbae183f6f2fd381 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:25.253201 23064 raft_consensus.cc:729] T 622197c9b8664d94bbae183f6f2fd381 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c688f3df56ac4b6d9b0c30fa4cc8ca2a, State: Initialized, Role: FOLLOWER
I20240503 02:00:25.253309 23064 consensus_queue.cc:260] T 622197c9b8664d94bbae183f6f2fd381 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:25.253626 23064 ts_tablet_manager.cc:1398] T 622197c9b8664d94bbae183f6f2fd381 P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:00:25.254891 7034 tablet_copy_service.cc:342] P 63331f455be049c1bdd6c23c59ac1bef: Request end of tablet copy session c688f3df56ac4b6d9b0c30fa4cc8ca2a-622197c9b8664d94bbae183f6f2fd381 received from {username='slave'} at 127.0.99.131:60361
I20240503 02:00:25.254993 7034 tablet_copy_service.cc:433] P 63331f455be049c1bdd6c23c59ac1bef: ending tablet copy session c688f3df56ac4b6d9b0c30fa4cc8ca2a-622197c9b8664d94bbae183f6f2fd381 on tablet 622197c9b8664d94bbae183f6f2fd381 with peer c688f3df56ac4b6d9b0c30fa4cc8ca2a
W20240503 02:00:25.266894 9692 consensus_peers.cc:485] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:25.267578 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:25.270464 6818 consensus_peers.cc:485] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:25.360452 7211 consensus_peers.cc:485] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:25.367285 9692 consensus_peers.cc:485] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:25.368415 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:25.369637 6949 consensus_peers.cc:485] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:25.385210 6949 consensus_peers.cc:485] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:25.397969 6949 consensus_peers.cc:485] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:25.414294 6818 consensus_peers.cc:485] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:25.428618 6949 consensus_peers.cc:485] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:25.434789 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:25.466204 9692 consensus_peers.cc:485] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:25.504858 22913 raft_consensus.cc:482] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:25.505043 22913 raft_consensus.cc:504] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:25.506240 22913 leader_election.cc:288] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:25.507598 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49eabbb8972840759ffa83951cb0b3b7" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
W20240503 02:00:25.507745 6816 leader_election.cc:334] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:25.507797 9756 raft_consensus.cc:2454] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
W20240503 02:00:25.508213 6818 leader_election.cc:334] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:25.508306 6818 leader_election.cc:302] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:25.508524 22913 raft_consensus.cc:2726] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:25.516433 22945 raft_consensus.cc:482] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:25.516602 22945 raft_consensus.cc:504] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:25.517103 22945 leader_election.cc:288] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:25.519623 7080 leader_election.cc:334] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:25.519768 7080 leader_election.cc:334] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:25.520033 7080 leader_election.cc:302] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:25.520406 21540 raft_consensus.cc:2726] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:25.552784 6818 consensus_peers.cc:485] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:25.554811 7211 consensus_peers.cc:485] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:25.582353 9692 consensus_peers.cc:485] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:25.642297 7080 consensus_peers.cc:485] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:25.646826 6949 consensus_peers.cc:485] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:25.711994 7209 consensus_peers.cc:485] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:25.751982 21540 raft_consensus.cc:482] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:25.752149 21540 raft_consensus.cc:504] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:25.752622 21540 leader_election.cc:288] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:25.754086 7078 leader_election.cc:334] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:25.754478 7080 leader_election.cc:334] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:25.754590 7080 leader_election.cc:302] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:25.754748 21540 raft_consensus.cc:2726] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:25.763269 7145 raft_consensus.cc:1203] T 622197c9b8664d94bbae183f6f2fd381 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240503 02:00:25.829458 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:25.874656 22914 raft_consensus.cc:482] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:25.874811 22914 raft_consensus.cc:504] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:25.875253 22914 leader_election.cc:288] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:25.878399 6949 leader_election.cc:334] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:25.878614 6949 leader_election.cc:334] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:25.878662 6949 leader_election.cc:302] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:25.878810 22914 raft_consensus.cc:2726] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:25.928342 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:25.934633 22562 raft_consensus.cc:482] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:25.934698 22913 raft_consensus.cc:482] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:25.934787 22562 raft_consensus.cc:504] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:25.934923 22913 raft_consensus.cc:504] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:25.935182 22562 leader_election.cc:288] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:25.935636 22913 leader_election.cc:288] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:25.937001 6816 leader_election.cc:334] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:25.937773 9692 leader_election.cc:334] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:25.937815 6818 leader_election.cc:334] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:25.937896 6818 leader_election.cc:302] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:25.938275 22913 raft_consensus.cc:2726] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:25.938710 9692 leader_election.cc:334] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:25.938954 9692 leader_election.cc:302] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:25.939332 22562 raft_consensus.cc:2726] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:25.940090 6947 consensus_peers.cc:485] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:25.942298 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:25.989476 22592 raft_consensus.cc:482] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:25.989641 22592 raft_consensus.cc:504] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:25.990283 22592 leader_election.cc:288] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:25.991622 7211 leader_election.cc:334] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:25.991914 7209 leader_election.cc:334] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:25.991993 7209 leader_election.cc:302] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:25.992255 22592 raft_consensus.cc:2726] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:26.001936 7211 consensus_peers.cc:485] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:26.023126 22945 raft_consensus.cc:482] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:26.023352 22945 raft_consensus.cc:504] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:26.023923 22945 leader_election.cc:288] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:26.027750 7080 leader_election.cc:334] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:26.027920 7080 leader_election.cc:334] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:26.027976 7080 leader_election.cc:302] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:26.028138 22945 raft_consensus.cc:2726] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:26.034272 9692 consensus_peers.cc:485] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.044571 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:26.060353 7080 consensus_peers.cc:485] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.061213 6818 consensus_peers.cc:485] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.077140 6816 consensus_peers.cc:485] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.098628 6949 consensus_peers.cc:485] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:26.103061 21176 raft_consensus.cc:482] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:26.103233 21176 raft_consensus.cc:504] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:26.103720 21176 leader_election.cc:288] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:26.105255 6818 leader_election.cc:334] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:26.106009 6816 leader_election.cc:334] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:26.106182 6816 leader_election.cc:302] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:26.106391 21176 raft_consensus.cc:2726] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:26.118846 6818 consensus_peers.cc:485] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.123102 6816 consensus_peers.cc:485] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.133395 6949 consensus_peers.cc:485] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.142668 9692 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.144146 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.144483 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.144553 6816 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.187917 6947 consensus_peers.cc:485] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.230330 7211 consensus_peers.cc:485] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.232134 7078 consensus_peers.cc:485] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.241698 7080 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.246037 6947 consensus_peers.cc:485] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.246356 9690 consensus_peers.cc:485] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.257472 7078 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.264907 7209 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.274868 7078 consensus_peers.cc:485] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.279266 7209 consensus_peers.cc:485] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.283294 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:26.288247 7078 consensus_peers.cc:485] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.289139 7080 consensus_peers.cc:485] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.290894 7078 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.317726 7078 consensus_peers.cc:485] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:26.323112 23074 raft_consensus.cc:482] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:26.323588 23074 raft_consensus.cc:504] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:26.324499 23074 leader_election.cc:288] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:26.325528 7080 consensus_peers.cc:485] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.326869 6949 leader_election.cc:334] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:26.326989 6949 leader_election.cc:334] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:26.327023 6949 leader_election.cc:302] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:26.327226 23074 raft_consensus.cc:2726] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:00:26.327770 7080 consensus_peers.cc:485] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.331286 6818 consensus_peers.cc:485] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.339470 6949 consensus_peers.cc:485] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.351030 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.357878 6947 consensus_peers.cc:485] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.362236 7211 consensus_peers.cc:485] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.365829 9692 consensus_peers.cc:485] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.370018 7078 consensus_peers.cc:485] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:26.394304 22913 raft_consensus.cc:482] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:26.394491 22913 raft_consensus.cc:504] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:26.394986 22913 leader_election.cc:288] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:26.395267 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:26.396435 6816 leader_election.cc:334] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:26.397064 6818 leader_election.cc:334] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:26.397224 6818 leader_election.cc:302] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:26.398003 21176 raft_consensus.cc:2726] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:26.398279 9692 consensus_peers.cc:485] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.409873 7209 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.410506 9690 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.420960 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.432601 6818 consensus_peers.cc:485] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.437213 6816 consensus_peers.cc:485] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.444861 6818 consensus_peers.cc:485] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.446998 9692 consensus_peers.cc:485] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.447201 7080 consensus_peers.cc:485] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.483836 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:26.487735 9692 consensus_peers.cc:485] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.488323 6949 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.504755 6818 consensus_peers.cc:485] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:26.505285 23074 raft_consensus.cc:482] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:26.505430 23074 raft_consensus.cc:504] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:26.506139 23074 leader_election.cc:288] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:26.508042 6949 leader_election.cc:334] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:26.508244 6949 leader_election.cc:334] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:26.508299 6949 leader_election.cc:302] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:26.508512 23074 raft_consensus.cc:2726] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:26.509606 9692 consensus_peers.cc:485] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.530618 7211 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.539000 7209 consensus_peers.cc:485] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.542378 6818 consensus_peers.cc:485] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:26.550020 21540 raft_consensus.cc:482] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:26.550182 21540 raft_consensus.cc:504] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:26.550751 21540 leader_election.cc:288] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:26.551991 7078 leader_election.cc:334] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:26.552117 7080 leader_election.cc:334] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:26.552186 7080 leader_election.cc:302] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:26.552333 21540 raft_consensus.cc:2726] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:26.566388 6816 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111) [suppressed 296 similar messages]
W20240503 02:00:26.569536 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111) [suppressed 221 similar messages]
W20240503 02:00:26.570585 7209 consensus_peers.cc:485] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.580250 6949 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111) [suppressed 292 similar messages]
W20240503 02:00:26.581022 7078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111) [suppressed 260 similar messages]
W20240503 02:00:26.582267 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.582886 6947 consensus_peers.cc:485] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.582995 7078 consensus_peers.cc:485] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.583164 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:26.586366 23074 raft_consensus.cc:482] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:26.586716 23074 raft_consensus.cc:504] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:26.587218 23074 leader_election.cc:288] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:26.588572 6947 leader_election.cc:334] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:26.588694 6949 leader_election.cc:334] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:26.588793 6949 leader_election.cc:302] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:26.588990 23074 raft_consensus.cc:2726] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:26.599637 6947 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.602258 6818 consensus_peers.cc:485] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.604053 7211 consensus_peers.cc:485] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.606961 6818 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.613809 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.614017 6816 consensus_peers.cc:485] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.614725 6818 consensus_peers.cc:485] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.631568 9690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111) [suppressed 216 similar messages]
W20240503 02:00:26.632297 9690 consensus_peers.cc:485] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.638235 7211 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.638846 6947 consensus_peers.cc:485] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:26.642645 23074 raft_consensus.cc:482] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:26.642860 23074 raft_consensus.cc:504] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:26.643352 23074 leader_election.cc:288] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:26.644659 6949 leader_election.cc:334] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:26.644788 6947 leader_election.cc:334] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:26.644850 6947 leader_election.cc:302] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:26.645040 23074 raft_consensus.cc:2726] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:26.664804 23074 raft_consensus.cc:482] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:26.664952 23074 raft_consensus.cc:504] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:26.665325 23074 leader_election.cc:288] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:26.666491 6949 leader_election.cc:334] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:26.666651 6949 leader_election.cc:334] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:26.666702 6949 leader_election.cc:302] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:26.667001 23074 raft_consensus.cc:2726] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:26.667956 22562 raft_consensus.cc:482] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:26.668073 22562 raft_consensus.cc:504] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:26.668470 22562 leader_election.cc:288] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:26.669656 9690 leader_election.cc:334] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:26.669749 9692 leader_election.cc:334] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:26.669824 9692 leader_election.cc:302] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:26.669979 22562 raft_consensus.cc:2726] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:26.673486 6949 consensus_peers.cc:485] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.679045 7078 consensus_peers.cc:485] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.680027 6818 consensus_peers.cc:485] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.680236 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:26.710201 22255 raft_consensus.cc:482] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:26.710335 22255 raft_consensus.cc:504] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:26.710958 22255 leader_election.cc:288] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:26.712218 9692 leader_election.cc:334] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:26.712345 9692 leader_election.cc:334] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:26.712383 9692 leader_election.cc:302] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:26.712678 22255 raft_consensus.cc:2726] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:26.714637 7080 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.720955 7078 consensus_peers.cc:485] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.727283 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.783339 7211 consensus_peers.cc:485] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.790478 9690 consensus_peers.cc:485] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.795595 6818 consensus_peers.cc:485] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.800771 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:26.807886 7209 consensus_peers.cc:485] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.813380 7211 consensus_peers.cc:485] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:26.814491 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:26.894423 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:26.906947 9690 consensus_peers.cc:485] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:26.915212 22255 raft_consensus.cc:482] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:26.915382 22255 raft_consensus.cc:504] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:26.916203 22255 leader_election.cc:288] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:26.917986 9692 leader_election.cc:334] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:26.918161 9692 leader_election.cc:334] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:26.918226 9692 leader_election.cc:302] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:26.918452 22255 raft_consensus.cc:2726] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:26.936997 22255 raft_consensus.cc:482] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:26.937280 22255 raft_consensus.cc:504] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:26.937770 22255 leader_election.cc:288] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:26.939705 9690 leader_election.cc:334] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:26.939867 9692 leader_election.cc:334] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:26.939982 9692 leader_election.cc:302] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:26.940240 22255 raft_consensus.cc:2726] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:26.976401 22592 raft_consensus.cc:482] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:26.976596 22592 raft_consensus.cc:504] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:26.977366 22592 leader_election.cc:288] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:26.979142 7211 leader_election.cc:334] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:26.979291 7211 leader_election.cc:334] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:26.979347 7211 leader_election.cc:302] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:26.979525 22592 raft_consensus.cc:2726] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:27.056031 22913 raft_consensus.cc:482] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:27.056191 22913 raft_consensus.cc:504] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:27.056638 22913 leader_election.cc:288] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:27.058679 6818 leader_election.cc:334] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:27.058934 6818 leader_election.cc:334] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:27.059120 6818 leader_election.cc:302] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:27.059331 22913 raft_consensus.cc:2726] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:27.077203 22945 raft_consensus.cc:482] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:27.077425 22945 raft_consensus.cc:504] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:27.078291 22945 leader_election.cc:288] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:27.080008 7078 leader_election.cc:334] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:27.080754 7080 leader_election.cc:334] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:27.080852 7080 leader_election.cc:302] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:27.081046 22945 raft_consensus.cc:2726] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:27.108234 22945 raft_consensus.cc:482] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:27.108677 22945 raft_consensus.cc:504] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:27.109218 22945 leader_election.cc:288] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:27.110718 7078 leader_election.cc:334] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:27.111419 7080 leader_election.cc:334] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:27.111492 7080 leader_election.cc:302] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
W20240503 02:00:27.111624 7080 leader_election.cc:334] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:27.111630 22945 raft_consensus.cc:2726] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:27.123248 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:00:27.163100 22913 raft_consensus.cc:482] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:27.163331 22913 raft_consensus.cc:504] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:27.164110 22913 leader_election.cc:288] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:27.165539 6816 leader_election.cc:334] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:27.165843 6818 leader_election.cc:334] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:27.165907 6818 leader_election.cc:302] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:27.166080 22913 raft_consensus.cc:2726] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:27.176551 22255 raft_consensus.cc:482] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:27.176703 22255 raft_consensus.cc:504] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:27.177511 22255 leader_election.cc:288] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:27.179222 9690 leader_election.cc:334] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:27.179389 9692 leader_election.cc:334] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:27.179452 9692 leader_election.cc:302] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:27.179802 22255 raft_consensus.cc:2726] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:27.190069 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:27.205662 22945 raft_consensus.cc:482] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:27.205850 22945 raft_consensus.cc:504] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:27.206370 22945 leader_election.cc:288] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:27.208168 7080 leader_election.cc:334] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:27.208652 7080 leader_election.cc:334] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:27.208725 7080 leader_election.cc:302] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:27.208971 22945 raft_consensus.cc:2726] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:27.306849 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:00:27.316774 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:27.379098 22945 raft_consensus.cc:482] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:27.379448 22945 raft_consensus.cc:504] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:27.380545 22945 leader_election.cc:288] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:27.381498 7080 leader_election.cc:334] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:27.382323 7078 leader_election.cc:334] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:27.382393 7078 leader_election.cc:302] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:27.382558 22945 raft_consensus.cc:2726] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:27.385404 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:27.624146 7209 consensus_peers.cc:485] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:27.638240 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:27.672281 6818 consensus_peers.cc:485] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:27.685137 9690 consensus_peers.cc:485] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:27.686851 7211 consensus_peers.cc:485] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:27.687738 22913 raft_consensus.cc:482] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:27.687853 22913 raft_consensus.cc:504] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:27.688405 22913 leader_election.cc:288] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:27.689270 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49eabbb8972840759ffa83951cb0b3b7" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
W20240503 02:00:27.689447 6816 leader_election.cc:334] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:27.689491 9757 raft_consensus.cc:2454] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
W20240503 02:00:27.690225 6818 leader_election.cc:334] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:27.690495 6818 leader_election.cc:302] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, db1a0000aaa346f79c0aa4b6c5105191; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:27.690919 22913 raft_consensus.cc:2726] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:27.777724 9692 consensus_peers.cc:485] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:27.787384 6949 consensus_peers.cc:485] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:27.812518 7211 consensus_peers.cc:485] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:27.816162 9692 consensus_peers.cc:485] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:27.826925 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:27.829491 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:27.876811 6818 consensus_peers.cc:485] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:27.889216 6949 consensus_peers.cc:485] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:27.920607 6818 consensus_peers.cc:485] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:27.964457 9692 consensus_peers.cc:485] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:27.998039 9692 consensus_peers.cc:485] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:28.012486 6818 consensus_peers.cc:485] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:28.038573 6949 consensus_peers.cc:485] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:28.114326 6949 consensus_peers.cc:485] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:28.119263 7209 consensus_peers.cc:485] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:28.156426 7080 consensus_peers.cc:485] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:28.183645 23082 raft_consensus.cc:482] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:28.183842 23082 raft_consensus.cc:504] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:28.184443 23082 leader_election.cc:288] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:28.186321 9692 leader_election.cc:334] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:28.186729 9692 leader_election.cc:334] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:28.186797 9692 leader_election.cc:302] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:28.187356 23082 raft_consensus.cc:2726] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:28.206811 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:28.214128 6947 consensus_peers.cc:485] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.272106 7078 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.290428 6949 consensus_peers.cc:485] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:28.290594 7211 consensus_peers.cc:485] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:28.366434 9692 consensus_peers.cc:485] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.375918 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.394723 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:28.401014 6818 consensus_peers.cc:485] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.440871 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:28.454859 9690 consensus_peers.cc:485] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.465576 7080 consensus_peers.cc:485] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.496846 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.505402 9692 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.519445 7209 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:28.547592 21540 raft_consensus.cc:482] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:28.547746 21540 raft_consensus.cc:504] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:28.548192 21540 leader_election.cc:288] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:28.549448 7080 leader_election.cc:334] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:28.549531 7078 leader_election.cc:334] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:28.549643 7078 leader_election.cc:302] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:28.549836 21540 raft_consensus.cc:2726] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:28.556107 6816 consensus_peers.cc:485] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.602646 6947 consensus_peers.cc:485] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.604200 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.606032 6816 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:28.628062 23076 raft_consensus.cc:482] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:28.628206 23076 raft_consensus.cc:504] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:28.628598 23076 leader_election.cc:288] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:28.630273 6949 leader_election.cc:334] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:28.630496 6949 leader_election.cc:334] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:28.630589 6949 leader_election.cc:302] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:28.630728 23076 raft_consensus.cc:2726] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:28.664520 6816 consensus_peers.cc:485] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.665933 6949 consensus_peers.cc:485] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.678211 7080 consensus_peers.cc:485] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.694870 6818 consensus_peers.cc:485] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.696905 6947 consensus_peers.cc:485] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.707535 7078 consensus_peers.cc:485] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.724866 7078 consensus_peers.cc:485] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.728138 6949 consensus_peers.cc:485] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.733036 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:28.733597 7078 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.746829 7080 consensus_peers.cc:485] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.749976 6818 consensus_peers.cc:485] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.769765 6818 consensus_peers.cc:485] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.770265 6947 consensus_peers.cc:485] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.775336 6947 consensus_peers.cc:485] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.786166 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:28.789027 7211 consensus_peers.cc:485] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.792577 7078 consensus_peers.cc:485] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.807471 7209 consensus_peers.cc:485] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:28.818578 21540 raft_consensus.cc:482] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:28.818719 21540 raft_consensus.cc:504] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:28.819172 21540 leader_election.cc:288] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:28.820724 7078 leader_election.cc:334] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:28.821098 7080 leader_election.cc:334] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:28.821218 7080 leader_election.cc:302] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:28.821403 21540 raft_consensus.cc:2726] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:28.824368 7078 consensus_peers.cc:485] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.828470 9692 consensus_peers.cc:485] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.832484 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.838888 6949 consensus_peers.cc:485] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.841197 7080 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.854430 7209 consensus_peers.cc:485] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.855599 6947 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.864132 6949 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:28.869132 22042 raft_consensus.cc:482] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:28.869280 22042 raft_consensus.cc:504] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:28.869709 22042 leader_election.cc:288] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:28.870970 7211 leader_election.cc:334] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:28.871095 7211 leader_election.cc:334] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:28.871173 7211 leader_election.cc:302] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:28.871347 22042 raft_consensus.cc:2726] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:28.873576 7211 consensus_peers.cc:485] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.899889 9692 consensus_peers.cc:485] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.919865 7209 consensus_peers.cc:485] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.940328 7209 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.945763 7078 consensus_peers.cc:485] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.958139 6818 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.959631 6818 consensus_peers.cc:485] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.963035 9692 consensus_peers.cc:485] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.964774 7080 consensus_peers.cc:485] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.964891 6818 consensus_peers.cc:485] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:28.965056 9690 consensus_peers.cc:485] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.009279 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:29.034425 7211 consensus_peers.cc:485] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.044440 6947 consensus_peers.cc:485] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.046437 7080 consensus_peers.cc:485] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.056419 7211 consensus_peers.cc:485] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.058252 9692 consensus_peers.cc:485] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.106168 9690 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.124629 7211 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.129302 7209 consensus_peers.cc:485] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.130923 6816 consensus_peers.cc:485] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.157209 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.166550 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.186661 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:29.190476 6818 consensus_peers.cc:485] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.201650 6818 consensus_peers.cc:485] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.201527 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.207340 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.223477 7211 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.226154 6818 consensus_peers.cc:485] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.232424 6949 consensus_peers.cc:485] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.237536 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.238814 6818 consensus_peers.cc:485] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.240509 6816 consensus_peers.cc:485] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.248438 9692 consensus_peers.cc:485] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:29.249742 23076 raft_consensus.cc:482] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:29.249886 23076 raft_consensus.cc:504] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:29.250296 23076 leader_election.cc:288] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:29.251413 6949 leader_election.cc:334] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:29.251531 6949 leader_election.cc:334] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:29.251581 6949 leader_election.cc:302] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:29.251706 23076 raft_consensus.cc:2726] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:29.260385 21540 raft_consensus.cc:482] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:29.260533 21540 raft_consensus.cc:504] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:29.260962 21540 leader_election.cc:288] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:29.262916 7080 leader_election.cc:334] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:29.263074 7080 leader_election.cc:334] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:29.263142 7080 leader_election.cc:302] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:29.263546 21540 raft_consensus.cc:2726] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:29.279970 23082 raft_consensus.cc:482] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:29.280210 23082 raft_consensus.cc:504] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:29.280750 23082 leader_election.cc:288] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:29.282037 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.282017 9690 leader_election.cc:334] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:29.282809 9692 leader_election.cc:334] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:29.282899 9692 leader_election.cc:302] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:29.283100 23082 raft_consensus.cc:2726] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:29.285473 7078 consensus_peers.cc:485] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.313571 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:29.352990 6818 consensus_peers.cc:485] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.367595 7078 consensus_peers.cc:485] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.367692 7211 consensus_peers.cc:485] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.381090 7211 consensus_peers.cc:485] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.394289 9690 consensus_peers.cc:485] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.395406 9690 consensus_peers.cc:485] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.416368 7078 consensus_peers.cc:485] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.445986 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:29.504151 23082 raft_consensus.cc:482] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:29.504289 23082 raft_consensus.cc:504] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:29.504691 23082 leader_election.cc:288] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:29.506290 9690 leader_election.cc:334] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:29.506943 9692 leader_election.cc:334] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:29.507023 9692 leader_election.cc:302] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:29.507229 23082 raft_consensus.cc:2726] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:29.562898 7080 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:29.591854 23089 raft_consensus.cc:482] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:29.591990 23089 raft_consensus.cc:504] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:29.592476 23089 leader_election.cc:288] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:29.593830 6816 leader_election.cc:334] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:29.594034 6818 leader_election.cc:334] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:29.594142 6818 leader_election.cc:302] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:29.594298 23089 raft_consensus.cc:2726] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:29.600818 21540 raft_consensus.cc:482] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:29.600983 21540 raft_consensus.cc:504] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:29.601481 21540 leader_election.cc:288] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:29.603302 7080 leader_election.cc:334] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:29.603819 7080 leader_election.cc:334] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:29.603893 7080 leader_election.cc:302] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:29.604046 21540 raft_consensus.cc:2726] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:29.675257 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:00:29.809669 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:29.909373 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:29.909775 6949 consensus_peers.cc:485] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:29.911479 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:29.932111 23082 raft_consensus.cc:482] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:29.932246 23082 raft_consensus.cc:504] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:29.932808 23082 leader_election.cc:288] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:29.933861 9692 leader_election.cc:334] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:29.933988 9692 leader_election.cc:334] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:29.934051 9692 leader_election.cc:302] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:29.934192 23082 raft_consensus.cc:2726] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:29.954789 6818 consensus_peers.cc:485] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:30.003050 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 02:00:30.076350 23086 raft_consensus.cc:482] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:30.076483 23086 raft_consensus.cc:504] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:30.076913 23086 leader_election.cc:288] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:30.078336 6949 leader_election.cc:334] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:30.078506 6949 leader_election.cc:334] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:30.078780 6949 leader_election.cc:302] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:30.078979 23086 raft_consensus.cc:2726] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:00:30.081182 9690 consensus_peers.cc:485] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:30.111244 22042 raft_consensus.cc:482] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:30.111410 22042 raft_consensus.cc:504] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:30.111901 22042 leader_election.cc:288] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:30.113071 7211 leader_election.cc:334] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:30.113265 7209 leader_election.cc:334] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:30.113333 7209 leader_election.cc:302] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:30.113468 22042 raft_consensus.cc:2726] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:30.200711 23082 raft_consensus.cc:482] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:30.200870 23082 raft_consensus.cc:504] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:30.201293 23082 leader_election.cc:288] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:30.202942 9692 leader_election.cc:334] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:30.203297 9692 leader_election.cc:334] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:30.203375 9692 leader_election.cc:302] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:30.203778 23082 raft_consensus.cc:2726] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:30.203897 23089 raft_consensus.cc:482] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:30.204006 23089 raft_consensus.cc:504] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:30.204491 23089 leader_election.cc:288] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:30.205334 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49eabbb8972840759ffa83951cb0b3b7" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 02:00:30.205520 9757 raft_consensus.cc:2454] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
W20240503 02:00:30.205621 6816 leader_election.cc:334] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:30.205791 6818 leader_election.cc:334] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:30.205940 6817 leader_election.cc:302] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, db1a0000aaa346f79c0aa4b6c5105191; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:30.206174 23089 raft_consensus.cc:2726] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:30.267614 7211 consensus_peers.cc:485] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:30.273828 7209 consensus_peers.cc:485] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:30.301002 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:30.316541 9692 consensus_peers.cc:485] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:30.329497 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:30.353487 9692 consensus_peers.cc:485] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:30.419957 6949 consensus_peers.cc:485] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:30.444305 6818 consensus_peers.cc:485] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:30.451036 6818 consensus_peers.cc:485] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:30.456115 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:30.458659 23089 raft_consensus.cc:482] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:30.458798 23089 raft_consensus.cc:504] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:30.459167 23089 leader_election.cc:288] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:30.460638 6816 leader_election.cc:334] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:30.460740 6818 leader_election.cc:334] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:30.460822 6818 leader_election.cc:302] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:30.461015 23089 raft_consensus.cc:2726] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:30.497303 6949 consensus_peers.cc:485] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:30.538834 23076 raft_consensus.cc:482] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:30.539047 23076 raft_consensus.cc:504] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:30.539541 23076 leader_election.cc:288] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:30.541450 6949 leader_election.cc:334] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:30.541576 6949 leader_election.cc:334] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:30.541643 6949 leader_election.cc:302] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:30.542150 23076 raft_consensus.cc:2726] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:30.544533 6949 consensus_peers.cc:485] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:30.594118 22592 raft_consensus.cc:482] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:30.594273 22592 raft_consensus.cc:504] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:30.594811 22592 leader_election.cc:288] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:30.596275 7211 leader_election.cc:334] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:30.596382 7211 leader_election.cc:334] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:30.596410 7211 leader_election.cc:302] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:30.596863 22592 raft_consensus.cc:2726] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:30.625007 23082 raft_consensus.cc:482] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:30.625174 23082 raft_consensus.cc:504] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:30.625651 23082 leader_election.cc:288] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:30.627355 9692 leader_election.cc:334] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:30.627506 9692 leader_election.cc:334] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:30.627563 9692 leader_election.cc:302] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:30.627934 23082 raft_consensus.cc:2726] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:30.632961 7211 consensus_peers.cc:485] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:30.639885 9692 consensus_peers.cc:485] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:30.657208 7078 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:30.669359 7209 consensus_peers.cc:485] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:30.693289 6818 consensus_peers.cc:485] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:30.719586 6947 consensus_peers.cc:485] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:30.762163 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:30.766960 23076 raft_consensus.cc:482] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:30.767097 23076 raft_consensus.cc:504] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:30.767508 23076 leader_election.cc:288] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:30.768760 6949 leader_election.cc:334] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:30.769326 6947 leader_election.cc:334] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:30.769425 6947 leader_election.cc:302] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:30.769595 23076 raft_consensus.cc:2726] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:30.771968 7080 consensus_peers.cc:485] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:30.817850 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:30.839849 9692 consensus_peers.cc:485] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:30.872999 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:30.875787 21540 raft_consensus.cc:482] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
W20240503 02:00:30.875880 6816 consensus_peers.cc:485] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:30.875928 21540 raft_consensus.cc:504] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:30.876324 21540 leader_election.cc:288] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:30.877899 7080 leader_election.cc:334] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:30.878023 7080 leader_election.cc:334] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:30.878072 7080 leader_election.cc:302] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:30.878259 21540 raft_consensus.cc:2726] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:30.884119 6949 consensus_peers.cc:485] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:30.888374 23076 raft_consensus.cc:482] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:30.888511 23076 raft_consensus.cc:504] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:30.888971 23076 leader_election.cc:288] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:30.890724 6947 leader_election.cc:334] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:30.891050 6949 leader_election.cc:334] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:30.891119 6949 leader_election.cc:302] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:30.891350 23076 raft_consensus.cc:2726] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:30.940641 9692 consensus_peers.cc:485] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:30.963356 7211 consensus_peers.cc:485] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:30.984041 7080 consensus_peers.cc:485] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.002295 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:31.061091 6816 consensus_peers.cc:485] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:31.061282 23099 raft_consensus.cc:482] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:31.061407 23099 raft_consensus.cc:504] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:31.061977 23099 leader_election.cc:288] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:31.063843 6818 leader_election.cc:334] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:31.064021 6816 leader_election.cc:334] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:31.064112 6816 leader_election.cc:302] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:31.064283 23099 raft_consensus.cc:2726] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:31.071636 6816 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.086728 6947 consensus_peers.cc:485] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.101714 6947 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.103305 6818 consensus_peers.cc:485] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.108417 6949 consensus_peers.cc:485] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.123569 7209 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.133670 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.134302 7080 consensus_peers.cc:485] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.163600 6947 consensus_peers.cc:485] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.168983 7080 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.172245 9692 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.183980 9690 consensus_peers.cc:485] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.199229 7078 consensus_peers.cc:485] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.200564 7078 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.210368 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:31.229725 6947 consensus_peers.cc:485] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.233796 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.255213 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:31.272204 7211 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.279587 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.285420 6818 consensus_peers.cc:485] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.289521 6947 consensus_peers.cc:485] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.290879 6818 consensus_peers.cc:485] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.307482 9690 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.319219 7078 consensus_peers.cc:485] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.331768 6949 consensus_peers.cc:485] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:31.332479 21540 raft_consensus.cc:482] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:31.332592 21540 raft_consensus.cc:504] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
W20240503 02:00:31.332962 7078 consensus_peers.cc:485] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:31.332988 21540 leader_election.cc:288] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:31.333890 7080 leader_election.cc:334] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:31.333967 7078 leader_election.cc:334] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:31.334028 7078 leader_election.cc:302] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:31.334148 21540 raft_consensus.cc:2726] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:31.336562 7078 consensus_peers.cc:485] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.339968 6818 consensus_peers.cc:485] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.344000 7211 consensus_peers.cc:485] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.352264 7211 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.358245 6818 consensus_peers.cc:485] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.361444 7078 consensus_peers.cc:485] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.369598 6818 consensus_peers.cc:485] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.392271 6949 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.396265 7209 consensus_peers.cc:485] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.400269 7209 consensus_peers.cc:485] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.411995 9692 consensus_peers.cc:485] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.421277 9692 consensus_peers.cc:485] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.428525 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:31.429174 21540 raft_consensus.cc:482] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:31.429289 21540 raft_consensus.cc:504] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:31.429744 21540 leader_election.cc:288] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:31.430961 7078 leader_election.cc:334] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:31.431143 7080 leader_election.cc:334] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:31.431205 7080 leader_election.cc:302] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:31.431346 23104 raft_consensus.cc:2726] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:31.436211 7211 consensus_peers.cc:485] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.443332 6949 consensus_peers.cc:485] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.454579 7080 consensus_peers.cc:485] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.466830 9692 consensus_peers.cc:485] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.467834 7080 consensus_peers.cc:485] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.475971 7209 consensus_peers.cc:485] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.492486 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.492772 7080 consensus_peers.cc:485] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.500020 6818 consensus_peers.cc:485] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.511270 6816 consensus_peers.cc:485] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.513185 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:31.527138 22255 raft_consensus.cc:482] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:31.527323 22255 raft_consensus.cc:504] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:31.527776 22255 leader_election.cc:288] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
I20240503 02:00:31.528244 22913 raft_consensus.cc:482] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:31.528405 22913 raft_consensus.cc:504] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:31.528939 22913 leader_election.cc:288] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:31.529240 9690 leader_election.cc:334] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:31.530416 9692 leader_election.cc:334] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:31.530503 9692 leader_election.cc:302] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
W20240503 02:00:31.530494 6818 leader_election.cc:334] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:31.530632 6818 leader_election.cc:334] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:31.530679 6818 leader_election.cc:302] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:31.530687 22255 raft_consensus.cc:2726] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:31.530804 22913 raft_consensus.cc:2726] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:31.531021 23104 raft_consensus.cc:482] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:31.531148 23104 raft_consensus.cc:504] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:31.531530 23104 leader_election.cc:288] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:31.533085 7080 leader_election.cc:334] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:31.533190 7078 leader_election.cc:334] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:31.533252 7078 leader_election.cc:302] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:31.533412 23104 raft_consensus.cc:2726] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:31.539227 6818 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.543828 7209 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.554814 6818 consensus_peers.cc:485] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.568114 7209 consensus_peers.cc:485] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.577991 6818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111) [suppressed 270 similar messages]
W20240503 02:00:31.579267 6818 consensus_peers.cc:485] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.584537 7211 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111) [suppressed 205 similar messages]
W20240503 02:00:31.584543 7078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111) [suppressed 225 similar messages]
W20240503 02:00:31.592839 7211 consensus_peers.cc:485] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.606199 6947 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111) [suppressed 270 similar messages]
W20240503 02:00:31.627693 6949 consensus_peers.cc:485] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.630934 6947 consensus_peers.cc:485] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.643682 9692 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111) [suppressed 190 similar messages]
I20240503 02:00:31.650847 23107 raft_consensus.cc:482] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:31.651137 23107 raft_consensus.cc:504] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:31.651616 23107 leader_election.cc:288] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:31.653393 6949 leader_election.cc:334] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:31.653513 6949 leader_election.cc:334] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:31.653565 6949 leader_election.cc:302] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:31.653750 23107 raft_consensus.cc:2726] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:31.663231 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:31.668871 7211 consensus_peers.cc:485] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.670792 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.677704 9692 consensus_peers.cc:485] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.681530 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.688293 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:31.710414 9690 consensus_peers.cc:485] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.718909 6816 consensus_peers.cc:485] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:31.752456 23104 raft_consensus.cc:482] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
W20240503 02:00:31.752840 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:31.752871 23104 raft_consensus.cc:504] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:31.753408 23104 leader_election.cc:288] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:31.754508 7080 leader_election.cc:334] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:31.754647 7078 leader_election.cc:334] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:31.754698 7078 leader_election.cc:302] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:31.754860 23104 raft_consensus.cc:2726] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:31.772622 7078 consensus_peers.cc:485] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.792660 6818 consensus_peers.cc:485] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.829926 9690 consensus_peers.cc:485] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.847458 9690 consensus_peers.cc:485] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:31.891248 23108 raft_consensus.cc:482] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:31.891377 23108 raft_consensus.cc:504] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:31.891901 23108 leader_election.cc:288] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:31.893028 7078 leader_election.cc:334] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:31.893108 7080 leader_election.cc:334] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:31.893183 7080 leader_election.cc:302] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:31.893342 23108 raft_consensus.cc:2726] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:31.893680 7080 leader_election.cc:334] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:31.910857 23089 raft_consensus.cc:482] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:31.911029 23089 raft_consensus.cc:504] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:31.911456 23089 leader_election.cc:288] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:31.912489 6818 leader_election.cc:334] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:31.912575 6816 leader_election.cc:334] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:31.912638 6816 leader_election.cc:302] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:31.912781 23089 raft_consensus.cc:2726] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:31.915661 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:31.932391 7078 consensus_peers.cc:485] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.934355 7078 consensus_peers.cc:485] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.948843 7080 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.960470 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:31.972056 7211 consensus_peers.cc:485] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:32.047991 9692 consensus_peers.cc:485] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:32.064785 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:32.070994 7211 consensus_peers.cc:485] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:32.141305 6818 consensus_peers.cc:485] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:32.162940 23108 raft_consensus.cc:482] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:32.163168 23108 raft_consensus.cc:504] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:32.164522 23108 leader_election.cc:288] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:32.165467 7080 leader_election.cc:334] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:32.165673 7080 leader_election.cc:334] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:32.165730 7080 leader_election.cc:302] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:32.165910 23108 raft_consensus.cc:2726] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:32.171679 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 02:00:32.198429 23089 raft_consensus.cc:482] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:32.198611 23089 raft_consensus.cc:504] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:32.199154 23089 leader_election.cc:288] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:32.200502 6816 leader_election.cc:334] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:32.200798 6818 leader_election.cc:334] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:32.200881 6818 leader_election.cc:302] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:32.201743 23089 raft_consensus.cc:2726] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:32.252633 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:32.325389 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:32.400753 23082 raft_consensus.cc:482] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:32.400946 23082 raft_consensus.cc:504] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:32.401636 23082 leader_election.cc:288] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 02:00:32.402223 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49eabbb8972840759ffa83951cb0b3b7" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 02:00:32.402448 6882 raft_consensus.cc:2454] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
W20240503 02:00:32.403313 9692 leader_election.cc:334] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:32.403791 9690 leader_election.cc:334] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:32.403914 9690 leader_election.cc:302] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, db1a0000aaa346f79c0aa4b6c5105191; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:32.404520 23082 raft_consensus.cc:2726] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:32.483079 6818 consensus_peers.cc:485] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:32.559695 6818 consensus_peers.cc:485] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:32.574234 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:32.582885 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:00:32.622012 6949 consensus_peers.cc:485] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:32.633119 7211 consensus_peers.cc:485] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:32.646301 9692 consensus_peers.cc:485] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:32.687186 23110 raft_consensus.cc:482] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:32.687325 23110 raft_consensus.cc:504] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:32.687747 23110 leader_election.cc:288] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:32.688951 6949 leader_election.cc:334] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:32.689079 6949 leader_election.cc:334] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:32.689136 6949 leader_election.cc:302] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:32.689280 23110 raft_consensus.cc:2726] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:00:32.720821 7209 consensus_peers.cc:485] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:32.808643 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:32.825543 9690 consensus_peers.cc:485] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:32.834950 9692 consensus_peers.cc:485] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:32.870278 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:32.888218 6818 consensus_peers.cc:485] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:32.922271 6949 consensus_peers.cc:485] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:32.956421 6949 consensus_peers.cc:485] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:32.958202 6949 consensus_peers.cc:485] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:32.974786 7211 consensus_peers.cc:485] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:33.019881 9692 consensus_peers.cc:485] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:33.038738 22255 raft_consensus.cc:482] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:33.038880 22255 raft_consensus.cc:504] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:33.039309 22255 leader_election.cc:288] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:33.040691 9690 leader_election.cc:334] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:33.041273 9692 leader_election.cc:334] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:33.041395 9692 leader_election.cc:302] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:33.041584 22255 raft_consensus.cc:2726] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:33.162712 6818 consensus_peers.cc:485] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:33.175401 7078 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.180320 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:33.196966 6947 consensus_peers.cc:485] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.225996 6949 consensus_peers.cc:485] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:33.228816 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.238723 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:33.259210 7209 consensus_peers.cc:485] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:33.367406 7080 consensus_peers.cc:485] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:33.378422 6816 consensus_peers.cc:485] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.388357 6818 consensus_peers.cc:485] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.389550 7209 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.402179 9692 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.429737 6816 consensus_peers.cc:485] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.463757 9690 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.470264 6816 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:33.473217 22255 raft_consensus.cc:482] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:33.473352 22255 raft_consensus.cc:504] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:33.473747 22255 leader_election.cc:288] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:33.474707 9690 leader_election.cc:334] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:33.474798 9692 leader_election.cc:334] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:33.474861 9692 leader_election.cc:302] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:33.474985 22255 raft_consensus.cc:2726] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:33.477772 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:33.486938 9692 consensus_peers.cc:485] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:33.514488 7080 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.525919 7078 consensus_peers.cc:485] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.525949 7080 consensus_peers.cc:485] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.527902 7209 consensus_peers.cc:485] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.537575 6947 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.540006 6818 consensus_peers.cc:485] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.566082 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.570101 7080 consensus_peers.cc:485] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.588896 9692 consensus_peers.cc:485] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.596405 7078 consensus_peers.cc:485] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.602820 7211 consensus_peers.cc:485] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:33.609118 6949 consensus_peers.cc:485] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.610061 6818 consensus_peers.cc:485] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.660725 7078 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.663754 7080 consensus_peers.cc:485] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.683869 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:33.688962 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.698448 6818 consensus_peers.cc:485] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.717603 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.754094 6816 consensus_peers.cc:485] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.768851 6947 consensus_peers.cc:485] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.778076 6947 consensus_peers.cc:485] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.780575 7211 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.787477 6947 consensus_peers.cc:485] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.787747 7211 consensus_peers.cc:485] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.789278 6818 consensus_peers.cc:485] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.798085 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:33.828179 6949 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.834041 9690 consensus_peers.cc:485] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.851248 7078 consensus_peers.cc:485] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.875171 6947 consensus_peers.cc:485] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.875499 7211 consensus_peers.cc:485] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.876413 9692 consensus_peers.cc:485] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.892297 6818 consensus_peers.cc:485] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.899737 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:33.917799 6818 consensus_peers.cc:485] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.931748 7209 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.946175 9692 consensus_peers.cc:485] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.977070 7209 consensus_peers.cc:485] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:33.979977 7211 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.008605 7078 consensus_peers.cc:485] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.011792 7080 consensus_peers.cc:485] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.019932 7209 consensus_peers.cc:485] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.024295 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.028159 6818 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.033121 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.040061 6818 consensus_peers.cc:485] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.041368 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.043538 7078 consensus_peers.cc:485] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.049283 9692 consensus_peers.cc:485] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.078833 6947 consensus_peers.cc:485] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.096172 6949 consensus_peers.cc:485] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.107457 6949 consensus_peers.cc:485] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.107656 9690 consensus_peers.cc:485] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.120085 7209 consensus_peers.cc:485] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.159562 7211 consensus_peers.cc:485] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.167239 7211 consensus_peers.cc:485] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.180969 7080 consensus_peers.cc:485] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.191015 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:34.194193 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.207818 9692 consensus_peers.cc:485] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.221869 7211 consensus_peers.cc:485] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:34.224429 23076 raft_consensus.cc:482] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:34.224586 23076 raft_consensus.cc:504] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:34.225029 23076 leader_election.cc:288] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:34.226492 6949 leader_election.cc:334] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:34.226692 6949 leader_election.cc:334] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:34.226742 6949 leader_election.cc:302] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:34.226960 23076 raft_consensus.cc:2726] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:34.245749 6818 consensus_peers.cc:485] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.248638 6818 consensus_peers.cc:485] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.270563 7078 consensus_peers.cc:485] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.288376 9692 consensus_peers.cc:485] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.303644 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:34.307852 23108 raft_consensus.cc:482] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:34.308022 23108 raft_consensus.cc:504] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:34.308466 23108 leader_election.cc:288] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:34.310318 7080 leader_election.cc:334] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:34.310492 7080 leader_election.cc:334] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:34.310730 7080 leader_election.cc:302] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:34.310914 23108 raft_consensus.cc:2726] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:34.327904 9690 consensus_peers.cc:485] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.341790 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:34.353639 6949 consensus_peers.cc:485] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.416882 7078 consensus_peers.cc:485] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.425782 9690 consensus_peers.cc:485] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:34.429390 23108 raft_consensus.cc:482] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:34.429574 23108 raft_consensus.cc:504] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:34.430228 23108 leader_election.cc:288] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:34.431424 7080 leader_election.cc:334] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:34.431679 7080 leader_election.cc:334] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:34.431787 7080 leader_election.cc:302] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:34.431998 23108 raft_consensus.cc:2726] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:34.439826 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.459367 6816 consensus_peers.cc:485] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.524385 7211 consensus_peers.cc:485] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.529107 7080 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.588250 7078 consensus_peers.cc:485] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.612110 6818 consensus_peers.cc:485] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.708766 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.749904 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:34.793190 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:34.829327 23110 raft_consensus.cc:482] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:34.829517 23110 raft_consensus.cc:504] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:34.830142 23110 leader_election.cc:288] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:34.831291 6949 leader_election.cc:334] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:34.831420 6949 leader_election.cc:334] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:34.831472 6949 leader_election.cc:302] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:34.831621 23110 raft_consensus.cc:2726] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:34.863400 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:34.901592 6818 consensus_peers.cc:485] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:34.907820 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 02:00:34.920296 22913 raft_consensus.cc:482] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:34.920446 22913 raft_consensus.cc:504] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:34.920955 22913 leader_election.cc:288] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:34.922506 6816 leader_election.cc:334] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:34.922734 6818 leader_election.cc:334] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:34.922819 6818 leader_election.cc:302] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:34.922998 22913 raft_consensus.cc:2726] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:34.948012 6818 consensus_peers.cc:485] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:35.064438 7211 consensus_peers.cc:485] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:35.074039 9692 consensus_peers.cc:485] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:35.109333 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:00:35.180668 6949 consensus_peers.cc:485] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:35.215049 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:35.228632 23089 raft_consensus.cc:482] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:35.228922 23089 raft_consensus.cc:504] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:35.229658 23089 leader_election.cc:288] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:35.230122 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49eabbb8972840759ffa83951cb0b3b7" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 02:00:35.230283 9757 raft_consensus.cc:2454] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
W20240503 02:00:35.231518 6816 leader_election.cc:334] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:35.231869 6818 leader_election.cc:334] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:35.231942 6818 leader_election.cc:302] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, db1a0000aaa346f79c0aa4b6c5105191; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:35.232146 23089 raft_consensus.cc:2726] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:35.238250 9692 consensus_peers.cc:485] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:35.270401 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:35.309012 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:35.315411 6949 consensus_peers.cc:485] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:35.327073 6949 consensus_peers.cc:485] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:35.358436 6818 consensus_peers.cc:485] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:35.378875 6818 consensus_peers.cc:485] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:35.391794 7209 consensus_peers.cc:485] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:35.394912 9690 consensus_peers.cc:485] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:35.444029 22255 raft_consensus.cc:482] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:35.444193 22255 raft_consensus.cc:504] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:35.444639 22255 leader_election.cc:288] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:35.445930 9692 leader_election.cc:334] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:35.446044 9692 leader_election.cc:334] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:35.446101 9692 leader_election.cc:302] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:35.446332 22255 raft_consensus.cc:2726] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:35.474042 7211 consensus_peers.cc:485] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:35.525053 6949 consensus_peers.cc:485] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:35.601464 9692 consensus_peers.cc:485] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:35.603600 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:35.656312 9692 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:35.667388 7209 consensus_peers.cc:485] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:35.674297 6949 consensus_peers.cc:485] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:35.685847 23076 raft_consensus.cc:482] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:35.686024 23076 raft_consensus.cc:504] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:35.686715 23076 leader_election.cc:288] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:35.687867 6949 leader_election.cc:334] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:35.688005 6949 leader_election.cc:334] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:35.688048 6949 leader_election.cc:302] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:35.688189 23076 raft_consensus.cc:2726] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:00:35.714327 6947 consensus_peers.cc:485] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:35.714538 6816 consensus_peers.cc:485] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:35.728804 7078 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:35.733377 7209 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:35.787066 6949 consensus_peers.cc:485] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:35.830605 6947 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:35.830919 6818 consensus_peers.cc:485] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:35.844168 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:35.850303 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:35.865345 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:35.881150 7080 consensus_peers.cc:485] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:35.897509 9690 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:35.937594 7078 consensus_peers.cc:485] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:35.962988 6818 consensus_peers.cc:485] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:35.965229 7078 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:35.968997 7080 consensus_peers.cc:485] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:35.980469 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:35.982220 7080 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:35.990381 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.028708 7080 consensus_peers.cc:485] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:36.029213 6947 consensus_peers.cc:485] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:36.034734 23104 raft_consensus.cc:482] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:36.034925 23104 raft_consensus.cc:504] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:36.035535 23104 leader_election.cc:288] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:36.036520 7078 leader_election.cc:334] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:36.037051 7080 leader_election.cc:334] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:36.037168 7080 leader_election.cc:302] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:36.037312 23104 raft_consensus.cc:2726] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:36.037322 7080 leader_election.cc:334] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:36.047240 9692 consensus_peers.cc:485] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:36.051566 6947 consensus_peers.cc:485] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.060699 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.068954 7209 consensus_peers.cc:485] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:36.109208 22913 raft_consensus.cc:482] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:36.109361 22913 raft_consensus.cc:504] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:36.109802 22913 leader_election.cc:288] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:36.111047 6818 leader_election.cc:334] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:36.111174 6818 leader_election.cc:334] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:36.111227 6818 leader_election.cc:302] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:36.111373 22913 raft_consensus.cc:2726] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:36.127305 23108 raft_consensus.cc:482] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:36.127457 23108 raft_consensus.cc:504] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:36.127933 23108 leader_election.cc:288] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:36.129416 7080 leader_election.cc:334] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:36.129559 7080 leader_election.cc:334] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:36.129602 7080 leader_election.cc:302] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:36.129745 23108 raft_consensus.cc:2726] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:36.132999 6816 consensus_peers.cc:485] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.148573 9692 consensus_peers.cc:485] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:36.152071 23076 raft_consensus.cc:482] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:36.152226 23076 raft_consensus.cc:504] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:36.152675 23076 leader_election.cc:288] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:36.153961 6947 leader_election.cc:334] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:36.154464 6949 leader_election.cc:334] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:36.154597 7078 consensus_peers.cc:485] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:36.154608 6949 leader_election.cc:302] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:36.155277 23076 raft_consensus.cc:2726] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:36.182420 6818 consensus_peers.cc:485] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:36.183233 23129 raft_consensus.cc:482] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:36.183362 23129 raft_consensus.cc:504] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:36.183988 23129 leader_election.cc:288] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:36.185390 9692 leader_election.cc:334] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:36.185796 9692 leader_election.cc:334] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:36.185866 9692 leader_election.cc:302] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:36.186100 22255 raft_consensus.cc:2726] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:36.188552 7211 consensus_peers.cc:485] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.194775 6816 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:36.206315 23076 raft_consensus.cc:482] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:36.206499 23076 raft_consensus.cc:504] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:36.206993 23076 leader_election.cc:288] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:36.208159 6949 leader_election.cc:334] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:36.208302 6949 leader_election.cc:334] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:36.208348 6949 leader_election.cc:302] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:36.208498 23076 raft_consensus.cc:2726] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:36.230000 7211 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.238554 7080 consensus_peers.cc:485] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.245147 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:36.251317 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.253269 6818 consensus_peers.cc:485] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.255309 9692 consensus_peers.cc:485] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.270200 7209 consensus_peers.cc:485] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.301708 7211 consensus_peers.cc:485] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.323112 6947 consensus_peers.cc:485] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.326035 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.336091 9692 consensus_peers.cc:485] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.337146 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.370769 7211 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.394994 6818 consensus_peers.cc:485] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.426877 7078 consensus_peers.cc:485] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.427249 6818 consensus_peers.cc:485] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.427325 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:36.428332 6818 consensus_peers.cc:485] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.430542 9692 consensus_peers.cc:485] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.430759 7078 consensus_peers.cc:485] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.460527 7209 consensus_peers.cc:485] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.468119 6818 consensus_peers.cc:485] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.495505 7211 consensus_peers.cc:485] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.511250 6818 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.515730 7080 consensus_peers.cc:485] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.518030 7211 consensus_peers.cc:485] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:36.518783 6949 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.526464 9690 consensus_peers.cc:485] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.543579 7209 consensus_peers.cc:485] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.545871 6818 consensus_peers.cc:485] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.546087 6816 consensus_peers.cc:485] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.548585 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.575517 6947 consensus_peers.cc:485] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.601696 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111) [suppressed 206 similar messages]
W20240503 02:00:36.602563 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:36.603474 22042 raft_consensus.cc:482] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:36.603718 22042 raft_consensus.cc:504] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:36.604199 22042 leader_election.cc:288] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:36.605345 7211 leader_election.cc:334] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:36.605525 6818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111) [suppressed 263 similar messages]
W20240503 02:00:36.605795 7209 leader_election.cc:334] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:36.605854 7209 leader_election.cc:302] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:36.605991 22042 raft_consensus.cc:2726] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:36.621300 7209 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.624814 6947 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111) [suppressed 264 similar messages]
W20240503 02:00:36.626714 7078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111) [suppressed 217 similar messages]
W20240503 02:00:36.626976 9690 consensus_peers.cc:485] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.627468 7078 consensus_peers.cc:485] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.648052 9692 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111) [suppressed 181 similar messages]
W20240503 02:00:36.655989 6949 consensus_peers.cc:485] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.679150 6947 consensus_peers.cc:485] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.698429 7078 consensus_peers.cc:485] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:36.703930 23108 raft_consensus.cc:482] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:36.704118 23108 raft_consensus.cc:504] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:36.704751 23108 leader_election.cc:288] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:36.706235 7080 leader_election.cc:334] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:36.706380 7080 leader_election.cc:334] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:36.706424 7080 leader_election.cc:302] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:36.706599 23108 raft_consensus.cc:2726] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:36.752630 7211 consensus_peers.cc:485] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.767467 7078 consensus_peers.cc:485] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.767689 9692 consensus_peers.cc:485] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.768850 6949 consensus_peers.cc:485] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.769208 7211 consensus_peers.cc:485] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.776005 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.781034 7080 consensus_peers.cc:485] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.787381 9690 consensus_peers.cc:485] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.792198 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:36.809665 7211 consensus_peers.cc:485] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.810115 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:36.819085 23129 raft_consensus.cc:482] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:36.819272 23129 raft_consensus.cc:504] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:36.819701 23129 leader_election.cc:288] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:36.821099 9692 leader_election.cc:334] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:36.821216 9692 leader_election.cc:334] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:36.821273 9692 leader_election.cc:302] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:36.821465 23129 raft_consensus.cc:2726] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:36.830401 9692 consensus_peers.cc:485] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.881233 6816 consensus_peers.cc:485] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.887866 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:36.909545 23104 raft_consensus.cc:482] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:36.909696 23104 raft_consensus.cc:504] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:36.910148 23104 leader_election.cc:288] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:36.911034 7078 leader_election.cc:334] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:36.911149 23104 raft_consensus.cc:482] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:36.911255 23104 raft_consensus.cc:504] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:36.911581 23104 leader_election.cc:288] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:36.912372 7078 leader_election.cc:334] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:36.912676 7080 leader_election.cc:334] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:36.912907 7080 leader_election.cc:302] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:36.913039 23104 raft_consensus.cc:2726] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:36.913717 7080 leader_election.cc:334] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:36.913781 7080 leader_election.cc:302] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:36.913887 23104 raft_consensus.cc:2726] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:36.940215 6949 consensus_peers.cc:485] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.975138 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:36.984756 6818 consensus_peers.cc:485] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:36.989841 6818 consensus_peers.cc:485] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:37.011946 9690 consensus_peers.cc:485] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:37.041263 7078 consensus_peers.cc:485] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:37.103782 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:37.107493 7080 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:37.197326 6818 consensus_peers.cc:485] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:37.230818 23110 raft_consensus.cc:482] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:37.230973 23110 raft_consensus.cc:504] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:37.231873 23110 leader_election.cc:288] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:37.233582 6949 leader_election.cc:334] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:37.233810 6949 leader_election.cc:334] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:37.233917 6949 leader_election.cc:302] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:37.234213 23110 raft_consensus.cc:2726] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:37.286352 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:37.351439 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:37.413385 22913 raft_consensus.cc:482] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:37.413512 22913 raft_consensus.cc:504] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:37.413905 22913 leader_election.cc:288] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:37.415136 6818 leader_election.cc:334] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:37.415280 6816 leader_election.cc:334] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:37.415341 6816 leader_election.cc:302] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:37.415757 22913 raft_consensus.cc:2726] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:37.457554 7211 consensus_peers.cc:485] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:37.506613 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:37.508431 6818 consensus_peers.cc:485] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:37.508780 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:00:37.515457 9692 consensus_peers.cc:485] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:37.603492 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 02:00:37.686599 23104 raft_consensus.cc:482] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:37.686849 23104 raft_consensus.cc:504] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:37.687363 23104 leader_election.cc:288] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:37.688611 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:37.688769 7080 leader_election.cc:334] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:37.689024 7078 leader_election.cc:334] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:37.689087 7078 leader_election.cc:302] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
W20240503 02:00:37.689337 7080 leader_election.cc:334] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:37.689419 23104 raft_consensus.cc:2726] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:37.712975 9690 consensus_peers.cc:485] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:37.714671 22255 raft_consensus.cc:482] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:37.714805 22255 raft_consensus.cc:504] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:37.715299 22255 leader_election.cc:288] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:37.716593 9692 leader_election.cc:334] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:37.716712 9692 leader_election.cc:334] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:37.716755 9692 leader_election.cc:302] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:37.716874 22255 raft_consensus.cc:2726] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:37.721850 9692 consensus_peers.cc:485] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:37.724854 6949 consensus_peers.cc:485] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:37.729926 6818 consensus_peers.cc:485] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:37.731890 7211 consensus_peers.cc:485] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 02:00:37.737965 23104 raft_consensus.cc:482] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:37.738199 23104 raft_consensus.cc:504] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:37.738945 23104 leader_election.cc:288] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:37.741472 7080 leader_election.cc:334] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:37.741627 7080 leader_election.cc:334] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:37.741688 7080 leader_election.cc:302] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:37.741840 23104 raft_consensus.cc:2726] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:37.752346 23110 raft_consensus.cc:482] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:37.752483 23110 raft_consensus.cc:504] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:37.752907 23110 leader_election.cc:288] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:37.753942 6949 leader_election.cc:334] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:37.754063 6947 leader_election.cc:334] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:37.754153 6947 leader_election.cc:302] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:37.754309 23110 raft_consensus.cc:2726] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:37.786068 6949 consensus_peers.cc:485] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:37.852471 6949 consensus_peers.cc:485] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:37.900812 7078 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:37.914078 23135 raft_consensus.cc:482] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:37.914206 23135 raft_consensus.cc:504] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:37.914625 23135 leader_election.cc:288] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:37.915657 6818 leader_election.cc:334] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:37.915804 6818 leader_election.cc:334] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:37.915850 6818 leader_election.cc:302] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:37.916035 23135 raft_consensus.cc:2726] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:37.928428 7209 consensus_peers.cc:485] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:37.931078 6949 consensus_peers.cc:485] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:37.933535 7209 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:37.937462 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:37.965516 7209 consensus_peers.cc:485] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:37.975215 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:37.981657 6816 consensus_peers.cc:485] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.026384 7080 consensus_peers.cc:485] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.087219 9692 consensus_peers.cc:485] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:38.114683 6818 consensus_peers.cc:485] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:38.127415 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:38.142413 7078 consensus_peers.cc:485] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.142740 6947 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.180050 6818 consensus_peers.cc:485] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.188257 6949 consensus_peers.cc:485] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.204161 9692 consensus_peers.cc:485] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:38.211508 6949 consensus_peers.cc:485] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:38.322360 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.324750 6947 consensus_peers.cc:485] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.328699 6818 consensus_peers.cc:485] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.361181 7080 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.363889 7078 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.367893 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:38.386657 9690 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.399837 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.408056 9692 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.440066 6816 consensus_peers.cc:485] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.478775 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:38.489003 7080 consensus_peers.cc:485] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:38.541085 7080 consensus_peers.cc:485] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.552558 7078 consensus_peers.cc:485] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.566056 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.588857 6947 consensus_peers.cc:485] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.613368 6947 consensus_peers.cc:485] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.625424 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.629089 7080 consensus_peers.cc:485] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.639364 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:38.643909 7209 consensus_peers.cc:485] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.644116 7211 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.651121 7211 consensus_peers.cc:485] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.673907 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.674757 6818 consensus_peers.cc:485] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.688274 6818 consensus_peers.cc:485] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.708573 6816 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.729971 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.775396 7211 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.788390 7211 consensus_peers.cc:485] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.799914 9692 consensus_peers.cc:485] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.813019 9692 consensus_peers.cc:485] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.819011 6818 consensus_peers.cc:485] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.828231 7209 consensus_peers.cc:485] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:38.850586 23135 raft_consensus.cc:482] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:38.850786 23135 raft_consensus.cc:504] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:38.851238 23135 leader_election.cc:288] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:38.852120 6818 leader_election.cc:334] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:38.852484 6816 leader_election.cc:334] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:38.852552 6816 leader_election.cc:302] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:38.852821 23135 raft_consensus.cc:2726] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:38.855044 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:38.859014 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:38.873952 23129 raft_consensus.cc:482] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:38.874162 23129 raft_consensus.cc:504] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:38.874788 23129 leader_election.cc:288] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:38.876029 9692 leader_election.cc:334] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:38.876224 9690 leader_election.cc:334] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:38.876312 9690 leader_election.cc:302] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:38.876487 23129 raft_consensus.cc:2726] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:38.877815 9690 consensus_peers.cc:485] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.920251 6818 consensus_peers.cc:485] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.920758 6818 consensus_peers.cc:485] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.938028 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:38.956843 9692 consensus_peers.cc:485] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.956851 7209 consensus_peers.cc:485] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.978590 7080 consensus_peers.cc:485] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:38.980437 9692 consensus_peers.cc:485] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:38.983361 23148 raft_consensus.cc:482] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:38.983530 23148 raft_consensus.cc:504] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:38.984514 23148 leader_election.cc:288] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:38.986093 6818 leader_election.cc:334] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:38.986336 6816 leader_election.cc:334] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:38.986415 6816 leader_election.cc:302] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:38.986586 23148 raft_consensus.cc:2726] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:38.987803 6947 consensus_peers.cc:485] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.010579 6949 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:39.026822 23143 raft_consensus.cc:482] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:39.026984 23143 raft_consensus.cc:504] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:39.027379 23143 leader_election.cc:288] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:39.028332 7080 leader_election.cc:334] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:39.028518 7078 leader_election.cc:334] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:39.028586 7078 leader_election.cc:302] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:39.028719 23143 raft_consensus.cc:2726] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:39.031428 6816 consensus_peers.cc:485] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.037569 6949 consensus_peers.cc:485] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.044458 6947 consensus_peers.cc:485] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.044862 7211 consensus_peers.cc:485] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.068166 7078 consensus_peers.cc:485] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.084373 6947 consensus_peers.cc:485] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.122629 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:39.138340 7080 consensus_peers.cc:485] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.145328 6949 consensus_peers.cc:485] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.148236 7078 consensus_peers.cc:485] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.148283 6818 consensus_peers.cc:485] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.152446 7078 consensus_peers.cc:485] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.169632 7209 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:39.174340 23142 raft_consensus.cc:482] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:39.174476 23142 raft_consensus.cc:504] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:39.174998 23142 leader_election.cc:288] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:39.176093 7080 leader_election.cc:334] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:39.176184 7078 leader_election.cc:334] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:39.176280 7078 leader_election.cc:302] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:39.176416 23142 raft_consensus.cc:2726] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:39.202931 6818 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.241917 7211 consensus_peers.cc:485] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:39.253885 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.281816 7078 consensus_peers.cc:485] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.282016 9690 consensus_peers.cc:485] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.293962 7211 consensus_peers.cc:485] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.301391 9690 consensus_peers.cc:485] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.320222 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:39.328985 6818 consensus_peers.cc:485] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.349079 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.349939 7211 consensus_peers.cc:485] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.359866 7078 consensus_peers.cc:485] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.382107 9692 consensus_peers.cc:485] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.385290 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:39.394207 9692 consensus_peers.cc:485] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.407624 6818 consensus_peers.cc:485] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.442312 6818 consensus_peers.cc:485] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.481123 7209 consensus_peers.cc:485] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.515990 7078 consensus_peers.cc:485] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.519767 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.529479 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:39.538058 7211 consensus_peers.cc:485] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.545457 7080 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.548835 6949 consensus_peers.cc:485] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.559836 9690 consensus_peers.cc:485] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:39.600631 23135 raft_consensus.cc:482] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:39.600798 23135 raft_consensus.cc:504] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:39.601330 23135 leader_election.cc:288] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:39.601923 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49eabbb8972840759ffa83951cb0b3b7" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 02:00:39.602131 9756 raft_consensus.cc:2454] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
W20240503 02:00:39.602689 6818 leader_election.cc:334] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:39.602872 6816 leader_election.cc:334] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:39.602926 6816 leader_election.cc:302] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, db1a0000aaa346f79c0aa4b6c5105191; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:39.603088 23135 raft_consensus.cc:2726] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:39.652657 6816 consensus_peers.cc:485] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240503 02:00:39.728756 23076 raft_consensus.cc:482] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:39.728919 23076 raft_consensus.cc:504] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:39.729332 23076 leader_election.cc:288] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:39.730612 6947 leader_election.cc:334] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:39.731098 6949 leader_election.cc:334] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:39.731213 6949 leader_election.cc:302] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:39.731377 23076 raft_consensus.cc:2726] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:39.782644 6818 consensus_peers.cc:485] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.831749 7211 consensus_peers.cc:485] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.889030 6818 consensus_peers.cc:485] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.911315 6949 consensus_peers.cc:485] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.944923 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:39.974804 6949 consensus_peers.cc:485] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:39.991140 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:40.011612 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:40.046805 7209 consensus_peers.cc:485] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:40.060391 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:40.065337 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:00:40.179649 6818 consensus_peers.cc:485] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:40.248876 9692 consensus_peers.cc:485] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:40.270962 7209 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:40.273912 6949 consensus_peers.cc:485] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:40.309401 6816 consensus_peers.cc:485] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:40.344321 9692 consensus_peers.cc:485] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:40.349056 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:40.349074 6949 consensus_peers.cc:485] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:40.350332 9690 consensus_peers.cc:485] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:40.377107 7078 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:40.445225 7209 consensus_peers.cc:485] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:40.511075 7211 consensus_peers.cc:485] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:40.527664 9692 consensus_peers.cc:485] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:40.549675 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:40.592522 6949 consensus_peers.cc:485] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:40.638370 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:40.678372 6947 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:40.684718 6818 consensus_peers.cc:485] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:40.695379 9692 consensus_peers.cc:485] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:40.773088 6947 consensus_peers.cc:485] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:40.774155 7078 consensus_peers.cc:485] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:40.785594 6818 consensus_peers.cc:485] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:40.786830 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:40.823038 6949 consensus_peers.cc:485] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:40.836396 7080 consensus_peers.cc:485] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:40.868487 7080 consensus_peers.cc:485] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:40.889297 7078 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:40.939915 7078 consensus_peers.cc:485] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:40.947892 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:40.953518 7080 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:40.961449 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:40.977651 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:40.988334 6816 consensus_peers.cc:485] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:40.997960 6816 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.014148 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:41.014197 7211 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.014360 6818 consensus_peers.cc:485] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.024513 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:41.032946 9692 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.073554 7080 consensus_peers.cc:485] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.094146 7080 consensus_peers.cc:485] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.114835 6818 consensus_peers.cc:485] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.137312 9690 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.147675 6947 consensus_peers.cc:485] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.149753 9692 consensus_peers.cc:485] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.158317 7211 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.180394 7209 consensus_peers.cc:485] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.191911 6947 consensus_peers.cc:485] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.197786 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.223527 6818 consensus_peers.cc:485] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.227957 9692 consensus_peers.cc:485] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.233031 6818 consensus_peers.cc:485] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.233565 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.234287 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.237107 6949 consensus_peers.cc:485] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.289755 6816 consensus_peers.cc:485] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.292522 7209 consensus_peers.cc:485] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.298573 6947 consensus_peers.cc:485] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.298689 6947 consensus_peers.cc:485] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.306212 7211 consensus_peers.cc:485] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.316393 7211 consensus_peers.cc:485] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.319061 7209 consensus_peers.cc:485] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.349318 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.352893 9692 consensus_peers.cc:485] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.370994 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:41.410394 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:41.423439 9692 consensus_peers.cc:485] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:00:41.425571 23142 raft_consensus.cc:482] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:41.425720 23142 raft_consensus.cc:504] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:41.427162 23142 leader_election.cc:288] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:41.428926 7078 leader_election.cc:334] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:41.429066 7080 leader_election.cc:334] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:41.429172 7080 leader_election.cc:302] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:41.429374 23142 raft_consensus.cc:2726] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:41.440568 6818 consensus_peers.cc:485] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.452560 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:41.491436 6818 consensus_peers.cc:485] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.507050 7080 consensus_peers.cc:485] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.507694 6818 consensus_peers.cc:485] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.508328 7211 consensus_peers.cc:485] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.548540 9690 consensus_peers.cc:485] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.555423 7078 consensus_peers.cc:485] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.579526 6949 consensus_peers.cc:485] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.597716 6949 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.617499 9690 consensus_peers.cc:485] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.617520 6818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111) [suppressed 259 similar messages]
W20240503 02:00:41.634696 6947 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111) [suppressed 260 similar messages]
W20240503 02:00:41.644336 7078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111) [suppressed 219 similar messages]
W20240503 02:00:41.657207 6947 consensus_peers.cc:485] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.697942 6818 consensus_peers.cc:485] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.698261 7211 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111) [suppressed 202 similar messages]
W20240503 02:00:41.699146 7211 consensus_peers.cc:485] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.721983 9692 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111) [suppressed 176 similar messages]
W20240503 02:00:41.726732 7080 consensus_peers.cc:485] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.726596 6818 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.729489 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.730240 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.797185 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:41.806142 7078 consensus_peers.cc:485] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:00:41.806773 23076 raft_consensus.cc:482] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:41.806947 23076 raft_consensus.cc:504] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:41.807400 23076 leader_election.cc:288] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:41.808933 6947 leader_election.cc:334] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:41.809031 6949 leader_election.cc:334] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:41.809083 6947 leader_election.cc:302] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:41.809231 23076 raft_consensus.cc:2726] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:41.836391 23168 raft_consensus.cc:482] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:41.836540 23168 raft_consensus.cc:504] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
W20240503 02:00:41.836833 7209 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:00:41.837227 23168 leader_election.cc:288] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 02:00:41.837952 6883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49eabbb8972840759ffa83951cb0b3b7" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 02:00:41.838124 6883 raft_consensus.cc:2454] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
W20240503 02:00:41.838163 9690 leader_election.cc:334] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:41.838505 9692 leader_election.cc:334] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:41.838732 9692 leader_election.cc:302] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, db1a0000aaa346f79c0aa4b6c5105191; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:41.839401 23168 raft_consensus.cc:2726] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:41.852416 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:00:41.853183 23168 raft_consensus.cc:482] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:41.853317 23168 raft_consensus.cc:504] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:41.853736 23168 leader_election.cc:288] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:41.854933 9690 leader_election.cc:334] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:41.855397 9692 leader_election.cc:334] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:41.855485 9692 leader_election.cc:302] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:41.855856 23168 raft_consensus.cc:2726] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:41.859416 7078 consensus_peers.cc:485] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.869869 7209 consensus_peers.cc:485] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.875588 6949 consensus_peers.cc:485] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.892239 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:41.907846 7078 consensus_peers.cc:485] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.908265 6818 consensus_peers.cc:485] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.909492 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:41.927731 9692 consensus_peers.cc:485] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.928797 7078 consensus_peers.cc:485] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 02:00:41.950556 23162 raft_consensus.cc:482] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:41.950861 23162 raft_consensus.cc:504] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:41.951628 23162 leader_election.cc:288] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:41.953475 7211 leader_election.cc:334] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:41.953663 7211 leader_election.cc:334] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:41.953724 7211 leader_election.cc:302] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:41.953951 23155 raft_consensus.cc:2726] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:41.958025 7211 consensus_peers.cc:485] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:41.960574 9690 consensus_peers.cc:485] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.986994 7211 consensus_peers.cc:485] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:41.995733 9690 consensus_peers.cc:485] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:42.001451 7080 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:42.013643 7078 consensus_peers.cc:485] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:42.034296 9692 consensus_peers.cc:485] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:42.108083 6818 consensus_peers.cc:485] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:42.209697 7211 consensus_peers.cc:485] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:42.257705 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:42.334381 6818 consensus_peers.cc:485] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:42.337283 6818 consensus_peers.cc:485] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:42.350044 6816 consensus_peers.cc:485] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:42.395206 6818 consensus_peers.cc:485] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:42.409509 7211 consensus_peers.cc:485] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:42.444043 6949 consensus_peers.cc:485] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:42.453927 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:42.458040 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:42.504410 7209 consensus_peers.cc:485] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:42.511826 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:42.550000 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:00:42.577446 6949 consensus_peers.cc:485] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:42.648646 7078 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:42.683825 9692 consensus_peers.cc:485] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:42.722618 6949 consensus_peers.cc:485] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:42.739913 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:42.760429 9692 consensus_peers.cc:485] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:42.762351 6949 consensus_peers.cc:485] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:42.801923 7209 consensus_peers.cc:485] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:42.843277 6816 consensus_peers.cc:485] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:42.859560 7209 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:42.890623 9690 consensus_peers.cc:485] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:42.961175 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:42.962666 23167 raft_consensus.cc:482] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:42.962807 23167 raft_consensus.cc:504] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:42.963696 23167 leader_election.cc:288] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:42.965214 7078 leader_election.cc:334] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:42.965303 7080 leader_election.cc:334] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:42.965368 7080 leader_election.cc:302] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:42.965507 23167 raft_consensus.cc:2726] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:42.978855 7211 consensus_peers.cc:485] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:42.986078 6949 consensus_peers.cc:485] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:43.061908 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:43.088366 6818 consensus_peers.cc:485] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:43.180189 6818 consensus_peers.cc:485] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.192046 7080 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.192469 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.211416 6947 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.228701 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:43.232692 23175 raft_consensus.cc:482] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:43.232837 23175 raft_consensus.cc:504] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:43.233282 23175 leader_election.cc:288] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:43.234489 6947 leader_election.cc:334] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:43.234619 6949 leader_election.cc:334] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:43.234905 6949 leader_election.cc:302] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:43.235100 23175 raft_consensus.cc:2726] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:43.241351 7078 consensus_peers.cc:485] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.267167 9692 consensus_peers.cc:485] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:43.322446 6816 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.367826 7080 consensus_peers.cc:485] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.368396 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:43.404098 9692 consensus_peers.cc:485] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:43.404207 7080 consensus_peers.cc:485] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.408038 7211 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.410642 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.426332 6816 consensus_peers.cc:485] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.431051 7080 consensus_peers.cc:485] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:43.459930 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:43.461125 9690 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.469182 7078 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.514883 6949 consensus_peers.cc:485] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.522949 6818 consensus_peers.cc:485] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.532456 7078 consensus_peers.cc:485] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:00:43.533156 23158 raft_consensus.cc:482] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:43.533270 23158 raft_consensus.cc:504] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:43.533694 23158 leader_election.cc:288] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:43.534662 9692 leader_election.cc:334] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:43.534862 9690 leader_election.cc:334] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:43.534920 9690 leader_election.cc:302] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:43.535059 23158 raft_consensus.cc:2726] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:43.546795 7080 consensus_peers.cc:485] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.548609 6949 consensus_peers.cc:485] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.569247 9692 consensus_peers.cc:485] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.574014 7211 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.591606 6947 consensus_peers.cc:485] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.592011 6818 consensus_peers.cc:485] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.618549 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.628234 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.668388 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.673060 7209 consensus_peers.cc:485] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.674652 6947 consensus_peers.cc:485] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.714452 6818 consensus_peers.cc:485] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.723659 9692 consensus_peers.cc:485] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.737473 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:43.743793 7209 consensus_peers.cc:485] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.748700 7211 consensus_peers.cc:485] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.752862 6818 consensus_peers.cc:485] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.756685 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:43.767700 6947 consensus_peers.cc:485] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.775324 6818 consensus_peers.cc:485] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.790942 6818 consensus_peers.cc:485] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.801668 6947 consensus_peers.cc:485] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.810127 7209 consensus_peers.cc:485] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.819320 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:00:43.832823 23168 raft_consensus.cc:482] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:43.832978 23168 raft_consensus.cc:504] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:43.833415 23168 leader_election.cc:288] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:43.834841 9692 leader_election.cc:334] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:43.834973 9692 leader_election.cc:334] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:43.835038 9692 leader_election.cc:302] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:43.835371 23168 raft_consensus.cc:2726] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:43.848604 6947 consensus_peers.cc:485] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.849368 7211 consensus_peers.cc:485] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.857945 6947 consensus_peers.cc:485] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.858069 9692 consensus_peers.cc:485] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.878948 7211 consensus_peers.cc:485] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.879732 9690 consensus_peers.cc:485] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.889086 7080 consensus_peers.cc:485] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.889484 9692 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.911818 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:43.929347 6818 consensus_peers.cc:485] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:43.944047 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.011297 9692 consensus_peers.cc:485] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.023748 6949 consensus_peers.cc:485] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.066488 6949 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.071944 6816 consensus_peers.cc:485] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.078023 7211 consensus_peers.cc:485] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.083954 7078 consensus_peers.cc:485] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.090545 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.184202 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:44.236249 7078 consensus_peers.cc:485] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.244921 7209 consensus_peers.cc:485] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.250640 6818 consensus_peers.cc:485] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.287492 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:44.295228 6949 consensus_peers.cc:485] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.301282 9692 consensus_peers.cc:485] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.327569 7211 consensus_peers.cc:485] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:44.341640 7209 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.355561 9692 consensus_peers.cc:485] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.373298 7078 consensus_peers.cc:485] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.387020 9690 consensus_peers.cc:485] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.395884 7078 consensus_peers.cc:485] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.403221 6818 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.408634 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.416483 7211 consensus_peers.cc:485] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.416901 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:44.444967 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.464479 9690 consensus_peers.cc:485] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.473014 7080 consensus_peers.cc:485] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.523540 7078 consensus_peers.cc:485] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.563146 9690 consensus_peers.cc:485] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.569383 7080 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.610829 6818 consensus_peers.cc:485] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.632850 7078 consensus_peers.cc:485] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.700181 6818 consensus_peers.cc:485] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.758898 6818 consensus_peers.cc:485] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.778205 7211 consensus_peers.cc:485] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.789255 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:44.798099 7211 consensus_peers.cc:485] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.819648 6816 consensus_peers.cc:485] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.839820 6949 consensus_peers.cc:485] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:00:44.900789 23148 raft_consensus.cc:482] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:44.901060 23148 raft_consensus.cc:504] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:44.901643 23148 leader_election.cc:288] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:44.904402 6818 leader_election.cc:334] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:44.904573 6818 leader_election.cc:334] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:44.904629 6818 leader_election.cc:302] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
W20240503 02:00:44.904799 6818 consensus_peers.cc:485] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:00:44.905112 23148 raft_consensus.cc:2726] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:44.907549 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:44.926501 6949 consensus_peers.cc:485] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:44.965639 7209 consensus_peers.cc:485] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:45.010144 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:45.034294 9692 consensus_peers.cc:485] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:45.081054 6818 consensus_peers.cc:485] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:45.143187 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:45.162990 6816 consensus_peers.cc:485] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:00:45.207578 23180 raft_consensus.cc:482] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:45.207746 23180 raft_consensus.cc:504] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:45.208410 23180 leader_election.cc:288] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 02:00:45.208937 6883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49eabbb8972840759ffa83951cb0b3b7" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 02:00:45.209128 6883 raft_consensus.cc:2454] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
W20240503 02:00:45.209599 9690 leader_election.cc:334] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:45.209998 9692 leader_election.cc:334] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:45.210067 9692 leader_election.cc:302] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, db1a0000aaa346f79c0aa4b6c5105191; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:45.210253 23180 raft_consensus.cc:2726] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:45.216032 9692 consensus_peers.cc:485] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:45.238305 6949 consensus_peers.cc:485] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:45.249379 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:45.311393 23178 raft_consensus.cc:482] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:45.311647 23178 raft_consensus.cc:504] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:45.312294 23178 leader_election.cc:288] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:45.313951 7211 leader_election.cc:334] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:45.314313 6949 consensus_peers.cc:485] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:45.317387 7209 leader_election.cc:334] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:45.317525 7209 leader_election.cc:302] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:45.317680 23178 raft_consensus.cc:2726] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:45.356659 7078 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:45.375846 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:45.390782 23181 raft_consensus.cc:482] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:45.390933 23181 raft_consensus.cc:504] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:45.391495 23181 leader_election.cc:288] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:45.392486 7078 leader_election.cc:334] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:45.393008 7080 leader_election.cc:334] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:45.393090 7080 leader_election.cc:302] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
W20240503 02:00:45.393251 7080 leader_election.cc:334] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:45.393311 23181 raft_consensus.cc:2726] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:45.397560 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:00:45.405117 7209 consensus_peers.cc:485] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:45.409930 9690 consensus_peers.cc:485] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:45.444950 6949 consensus_peers.cc:485] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:00:45.454242 23142 raft_consensus.cc:482] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:45.454506 23142 raft_consensus.cc:504] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:45.455546 23142 leader_election.cc:288] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:45.456552 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:45.457051 7080 leader_election.cc:334] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:45.457198 7078 leader_election.cc:334] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:45.457293 7078 leader_election.cc:302] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:45.457453 23142 raft_consensus.cc:2726] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:45.498450 23168 raft_consensus.cc:482] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:45.498648 23168 raft_consensus.cc:504] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:45.499137 23168 leader_election.cc:288] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:45.500766 9692 leader_election.cc:334] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:45.501045 9692 leader_election.cc:334] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:45.501117 9692 leader_election.cc:302] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:45.501266 23168 raft_consensus.cc:2726] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:45.515097 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:45.587499 7209 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:45.629540 7080 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:45.651036 7211 consensus_peers.cc:485] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:45.698776 9690 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:45.704691 6818 consensus_peers.cc:485] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:45.716591 9692 consensus_peers.cc:485] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:45.723207 6818 consensus_peers.cc:485] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:45.786317 7080 consensus_peers.cc:485] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 02:00:45.803366 23168 raft_consensus.cc:482] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:45.803516 23168 raft_consensus.cc:504] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:45.803964 23168 leader_election.cc:288] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:45.805289 9690 leader_election.cc:334] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:45.805390 9692 leader_election.cc:334] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:45.805475 9692 leader_election.cc:302] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:45.805641 23168 raft_consensus.cc:2726] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:45.807608 6816 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:00:45.819121 23186 raft_consensus.cc:482] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:45.819375 23186 raft_consensus.cc:504] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:45.819919 23186 leader_election.cc:288] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:45.820641 7080 consensus_peers.cc:485] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:45.821748 6949 leader_election.cc:334] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:45.821913 6949 leader_election.cc:334] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:45.821970 6949 leader_election.cc:302] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:45.822149 23186 raft_consensus.cc:2726] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:45.835656 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:45.838703 9692 consensus_peers.cc:485] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:45.877349 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:45.892467 7078 consensus_peers.cc:485] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:45.901198 6947 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:00:45.901906 23169 raft_consensus.cc:482] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:45.902048 23169 raft_consensus.cc:504] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:45.902822 23169 leader_election.cc:288] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:45.903719 6816 leader_election.cc:334] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:45.905120 6818 leader_election.cc:334] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:45.905193 6818 leader_election.cc:302] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:45.905427 23169 raft_consensus.cc:2726] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:45.933842 7080 consensus_peers.cc:485] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:45.947819 7078 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:45.950105 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:45.952778 6816 consensus_peers.cc:485] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:45.952792 6818 consensus_peers.cc:485] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:45.960726 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:45.966495 7078 consensus_peers.cc:485] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:45.981752 6818 consensus_peers.cc:485] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:45.998135 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.000074 6949 consensus_peers.cc:485] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.001960 6947 consensus_peers.cc:485] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.002255 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:46.006737 6949 consensus_peers.cc:485] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.006814 6947 consensus_peers.cc:485] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.058292 6818 consensus_peers.cc:485] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.069676 7211 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.087903 7211 consensus_peers.cc:485] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.089367 6818 consensus_peers.cc:485] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.089493 6818 consensus_peers.cc:485] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.110280 7080 consensus_peers.cc:485] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.174185 9692 consensus_peers.cc:485] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.175004 7209 consensus_peers.cc:485] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.184105 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.217639 6947 consensus_peers.cc:485] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.218246 9690 consensus_peers.cc:485] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.280193 6947 consensus_peers.cc:485] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.290583 7211 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.305356 6949 consensus_peers.cc:485] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.307554 9692 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.311506 6818 consensus_peers.cc:485] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.316013 9692 consensus_peers.cc:485] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.323184 7080 consensus_peers.cc:485] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.327105 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.333683 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:46.367996 6947 consensus_peers.cc:485] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.380714 7209 consensus_peers.cc:485] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.446192 6947 consensus_peers.cc:485] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.463605 6816 consensus_peers.cc:485] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.469429 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.477654 7211 consensus_peers.cc:485] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.494803 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:46.499104 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:46.499900 7209 consensus_peers.cc:485] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.510329 7211 consensus_peers.cc:485] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.510454 6818 consensus_peers.cc:485] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.517449 6949 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:00:46.521145 23181 raft_consensus.cc:482] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:46.521299 23181 raft_consensus.cc:504] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:46.521847 23181 leader_election.cc:288] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:46.525501 7080 leader_election.cc:334] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:46.525630 7080 leader_election.cc:334] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:46.525672 7080 leader_election.cc:302] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:46.525821 23181 raft_consensus.cc:2726] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:46.535836 7209 consensus_peers.cc:485] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.551477 7078 consensus_peers.cc:485] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.556922 6949 consensus_peers.cc:485] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.564759 9690 consensus_peers.cc:485] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.577248 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.609803 7211 consensus_peers.cc:485] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.619769 6818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111) [suppressed 255 similar messages]
W20240503 02:00:46.668421 6949 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111) [suppressed 259 similar messages]
W20240503 02:00:46.680989 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.690330 6818 consensus_peers.cc:485] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.693830 7080 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111) [suppressed 208 similar messages]
W20240503 02:00:46.700549 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111) [suppressed 204 similar messages]
W20240503 02:00:46.702667 7211 consensus_peers.cc:485] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.721532 9692 consensus_peers.cc:485] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.738677 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.758054 9692 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111) [suppressed 186 similar messages]
W20240503 02:00:46.758809 9692 consensus_peers.cc:485] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.784425 7078 consensus_peers.cc:485] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.832890 7078 consensus_peers.cc:485] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.891301 9692 consensus_peers.cc:485] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.926615 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:46.936216 7209 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.937265 7078 consensus_peers.cc:485] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.957201 6818 consensus_peers.cc:485] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:46.963832 7211 consensus_peers.cc:485] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:46.981832 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:47.046921 9692 consensus_peers.cc:485] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:47.066234 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:47.093884 6818 consensus_peers.cc:485] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:47.122399 7080 consensus_peers.cc:485] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:47.173727 9690 consensus_peers.cc:485] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:47.181593 6818 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:47.183151 7078 consensus_peers.cc:485] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:47.193228 9690 consensus_peers.cc:485] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:47.213166 7211 consensus_peers.cc:485] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:47.270210 7080 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:47.325091 6818 consensus_peers.cc:485] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:47.327133 7078 consensus_peers.cc:485] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:47.334383 6816 consensus_peers.cc:485] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:47.334542 6818 consensus_peers.cc:485] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:47.369539 6949 consensus_peers.cc:485] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:47.370044 6949 consensus_peers.cc:485] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:47.378170 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:47.399554 23175 raft_consensus.cc:482] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:47.399693 23175 raft_consensus.cc:504] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:47.400511 23175 leader_election.cc:288] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:47.401887 6949 leader_election.cc:334] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:47.402061 6949 leader_election.cc:334] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:47.402114 6949 leader_election.cc:302] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:47.402268 23186 raft_consensus.cc:2726] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:00:47.437423 6818 consensus_peers.cc:485] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:00:47.463546 23169 raft_consensus.cc:482] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:47.463699 23169 raft_consensus.cc:504] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:47.464295 23169 leader_election.cc:288] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:47.465873 6818 leader_election.cc:334] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:47.466009 6816 leader_election.cc:334] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:47.466187 6816 leader_election.cc:302] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:47.466679 23169 raft_consensus.cc:2726] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:47.484795 7211 consensus_peers.cc:485] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:47.505340 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:47.539111 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:47.558094 9690 consensus_peers.cc:485] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:47.579106 7209 consensus_peers.cc:485] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:47.616122 9692 consensus_peers.cc:485] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:00:47.618562 23180 raft_consensus.cc:482] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:47.618680 23180 raft_consensus.cc:504] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:47.619031 23180 leader_election.cc:288] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:47.620273 9692 leader_election.cc:334] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:47.620398 9692 leader_election.cc:334] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:47.620440 9692 leader_election.cc:302] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:47.620604 23180 raft_consensus.cc:2726] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:47.686439 6816 consensus_peers.cc:485] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:47.727980 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:47.739755 7078 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:47.771176 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:47.797564 6949 consensus_peers.cc:485] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:47.803584 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:47.860488 6949 consensus_peers.cc:485] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:47.884953 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:00:47.905447 9692 consensus_peers.cc:485] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:47.919631 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:47.954233 6949 consensus_peers.cc:485] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:47.967250 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:47.993800 7209 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.027379 6818 consensus_peers.cc:485] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.057803 6818 consensus_peers.cc:485] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:48.093817 9690 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.108940 7209 consensus_peers.cc:485] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:00:48.152850 23142 raft_consensus.cc:482] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:48.153002 23142 raft_consensus.cc:504] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:48.153508 23142 leader_election.cc:288] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:48.154868 7080 leader_election.cc:334] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:48.155009 7078 leader_election.cc:334] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:48.155081 7078 leader_election.cc:302] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:48.155212 23142 raft_consensus.cc:2726] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:48.155671 9692 consensus_peers.cc:485] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:48.168849 7211 consensus_peers.cc:485] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:00:48.179818 23142 raft_consensus.cc:482] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:48.179981 23142 raft_consensus.cc:504] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:48.180452 23142 leader_election.cc:288] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:48.181726 7078 leader_election.cc:334] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:48.181819 7080 leader_election.cc:334] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:48.181903 7080 leader_election.cc:302] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:48.182121 23142 raft_consensus.cc:2726] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:48.183271 6816 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.195166 7080 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.205343 7078 consensus_peers.cc:485] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:00:48.214159 23142 raft_consensus.cc:482] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:48.214293 23142 raft_consensus.cc:504] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:48.214788 23142 leader_election.cc:288] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:48.216125 7080 leader_election.cc:334] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:48.216244 7080 leader_election.cc:334] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:48.216296 7080 leader_election.cc:302] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:48.216765 23181 raft_consensus.cc:2726] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:48.224390 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:48.242817 7080 consensus_peers.cc:485] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.255792 6816 consensus_peers.cc:485] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.257797 6818 consensus_peers.cc:485] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:00:48.306017 23169 raft_consensus.cc:482] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:48.306161 23169 raft_consensus.cc:504] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:48.306645 23169 leader_election.cc:288] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:48.307723 6818 leader_election.cc:334] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:48.308010 6816 leader_election.cc:334] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:48.308069 6816 leader_election.cc:302] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:48.308199 23169 raft_consensus.cc:2726] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:48.313925 6947 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.330905 7080 consensus_peers.cc:485] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:48.341691 6949 consensus_peers.cc:485] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.341977 9692 consensus_peers.cc:485] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:48.367477 7078 consensus_peers.cc:485] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.378469 7080 consensus_peers.cc:485] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.391824 6818 consensus_peers.cc:485] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.393417 6818 consensus_peers.cc:485] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.442190 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.442322 7211 consensus_peers.cc:485] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.453053 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:48.496634 7080 consensus_peers.cc:485] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.506785 6947 consensus_peers.cc:485] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.510996 6947 consensus_peers.cc:485] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.534422 7078 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.536854 6818 consensus_peers.cc:485] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:00:48.537467 23186 raft_consensus.cc:482] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:48.537575 23186 raft_consensus.cc:504] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:48.538009 23186 leader_election.cc:288] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:48.538976 6949 leader_election.cc:334] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:48.539067 6947 leader_election.cc:334] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:48.539152 6947 leader_election.cc:302] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:48.539286 23186 raft_consensus.cc:2726] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:48.563452 9690 consensus_peers.cc:485] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.579298 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:48.590834 9692 consensus_peers.cc:485] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.593078 7211 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.627537 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.645056 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:48.664777 7209 consensus_peers.cc:485] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:00:48.692051 23142 raft_consensus.cc:482] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:48.692200 23142 raft_consensus.cc:504] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:48.692664 23142 leader_election.cc:288] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:48.693776 7080 leader_election.cc:334] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:48.693910 7080 leader_election.cc:334] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:48.693956 7080 leader_election.cc:302] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:48.694185 23142 raft_consensus.cc:2726] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:48.708106 6949 consensus_peers.cc:485] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.744582 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.751962 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.772070 7209 consensus_peers.cc:485] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.783375 6816 consensus_peers.cc:485] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.789371 6949 consensus_peers.cc:485] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.805510 6947 consensus_peers.cc:485] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.838337 7211 consensus_peers.cc:485] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.858970 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.865010 6947 consensus_peers.cc:485] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.872035 6818 consensus_peers.cc:485] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.872557 7211 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:00:48.875190 23180 raft_consensus.cc:482] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:48.875344 23180 raft_consensus.cc:504] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
W20240503 02:00:48.875620 6949 consensus_peers.cc:485] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:00:48.875805 23180 leader_election.cc:288] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:48.876765 9690 leader_election.cc:334] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:48.876953 9692 leader_election.cc:334] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:48.877030 9692 leader_election.cc:302] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:48.877171 23180 raft_consensus.cc:2726] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:48.880951 23169 raft_consensus.cc:482] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:48.881072 23169 raft_consensus.cc:504] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:48.881528 23169 leader_election.cc:288] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:48.883213 6818 leader_election.cc:334] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:48.883350 6818 leader_election.cc:334] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:48.883383 6818 leader_election.cc:302] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:48.883514 23169 raft_consensus.cc:2726] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:48.895989 7080 consensus_peers.cc:485] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:00:48.923918 23186 raft_consensus.cc:482] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:48.924141 23186 raft_consensus.cc:504] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:48.924862 23186 leader_election.cc:288] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:48.927207 6949 leader_election.cc:334] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:48.927331 6949 leader_election.cc:334] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:48.927390 6949 leader_election.cc:302] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:48.927565 23186 raft_consensus.cc:2726] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:48.932317 7078 consensus_peers.cc:485] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.944262 9692 consensus_peers.cc:485] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.960960 6947 consensus_peers.cc:485] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:48.964159 7211 consensus_peers.cc:485] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.002849 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:49.004072 6818 consensus_peers.cc:485] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.013224 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:49.018352 7209 consensus_peers.cc:485] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.024137 6947 consensus_peers.cc:485] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.050719 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.097553 6818 consensus_peers.cc:485] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.125720 6949 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.129215 9692 consensus_peers.cc:485] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.135133 7209 consensus_peers.cc:485] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.146989 7078 consensus_peers.cc:485] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.147094 7078 consensus_peers.cc:485] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.175051 9690 consensus_peers.cc:485] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.178954 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:49.220101 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.221156 6818 consensus_peers.cc:485] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.226775 9692 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.247328 7211 consensus_peers.cc:485] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.260826 9692 consensus_peers.cc:485] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.282824 7209 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.309098 9692 consensus_peers.cc:485] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.315304 7078 consensus_peers.cc:485] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.317698 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.375751 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.437510 7211 consensus_peers.cc:485] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:49.457386 6818 consensus_peers.cc:485] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.471740 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:49.501132 7211 consensus_peers.cc:485] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.531581 9690 consensus_peers.cc:485] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.554948 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:49.578132 23169 raft_consensus.cc:482] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:00:49.578287 23169 raft_consensus.cc:504] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:49.578886 23169 leader_election.cc:288] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:00:49.579254 9757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49eabbb8972840759ffa83951cb0b3b7" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 02:00:49.579429 9757 raft_consensus.cc:2454] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
W20240503 02:00:49.579950 6816 leader_election.cc:334] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:49.580161 6818 leader_election.cc:334] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:49.580224 6818 leader_election.cc:302] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, db1a0000aaa346f79c0aa4b6c5105191; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:49.580367 23169 raft_consensus.cc:2726] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:49.589535 7080 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.616725 6818 consensus_peers.cc:485] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.630040 6818 consensus_peers.cc:485] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.641815 7078 consensus_peers.cc:485] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.685501 6818 consensus_peers.cc:485] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.735394 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:49.768882 6949 consensus_peers.cc:485] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.793395 6818 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.830579 7211 consensus_peers.cc:485] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.831581 9692 consensus_peers.cc:485] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.835436 9690 consensus_peers.cc:485] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.867656 7211 consensus_peers.cc:485] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.883564 6818 consensus_peers.cc:485] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:00:49.914113 23209 raft_consensus.cc:482] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:49.914268 23209 raft_consensus.cc:504] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:49.914793 23209 leader_election.cc:288] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:49.916146 7080 leader_election.cc:334] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:49.916234 7078 leader_election.cc:334] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:49.916296 7078 leader_election.cc:302] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:49.916472 23209 raft_consensus.cc:2726] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:49.927479 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:00:49.934365 7080 consensus_peers.cc:485] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:49.936898 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:49.956440 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:50.013202 6949 consensus_peers.cc:485] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:50.046871 7078 consensus_peers.cc:485] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:50.059891 6816 consensus_peers.cc:485] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:50.092052 9690 consensus_peers.cc:485] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:50.112783 7209 consensus_peers.cc:485] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:50.178143 9692 consensus_peers.cc:485] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:50.203385 9692 consensus_peers.cc:485] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:50.233264 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:50.268404 6816 consensus_peers.cc:485] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 02:00:50.276131 23186 raft_consensus.cc:482] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:50.276297 23186 raft_consensus.cc:504] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:50.276731 23186 leader_election.cc:288] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:50.277953 6949 leader_election.cc:334] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:50.278115 6949 leader_election.cc:334] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:50.278173 6949 leader_election.cc:302] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:50.278298 23186 raft_consensus.cc:2726] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:50.287513 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:50.293573 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240503 02:00:50.307406 23142 raft_consensus.cc:482] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:50.307549 23142 raft_consensus.cc:504] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:50.308008 23142 leader_election.cc:288] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:50.309422 7080 leader_election.cc:334] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:50.309554 7080 leader_election.cc:334] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:50.309607 7080 leader_election.cc:302] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:50.309765 23142 raft_consensus.cc:2726] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:50.344964 6949 consensus_peers.cc:485] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:50.351037 6818 consensus_peers.cc:485] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:50.384581 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:50.403641 7078 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:50.449489 6949 consensus_peers.cc:485] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:50.468189 6949 consensus_peers.cc:485] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:50.499914 7078 consensus_peers.cc:485] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:50.500478 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:50.534444 6818 consensus_peers.cc:485] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:50.542429 7080 consensus_peers.cc:485] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:50.558584 7209 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:50.567399 7211 consensus_peers.cc:485] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:50.649582 7209 consensus_peers.cc:485] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:50.667152 7080 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:50.719619 9690 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:00:50.727550 23169 raft_consensus.cc:482] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:50.727713 23169 raft_consensus.cc:504] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:50.728433 23169 leader_election.cc:288] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:50.729568 6818 consensus_peers.cc:485] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:50.729665 7078 consensus_peers.cc:485] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:50.730397 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:50.730937 6818 leader_election.cc:334] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:50.731798 6818 consensus_peers.cc:485] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:50.732331 6816 leader_election.cc:334] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:50.732404 6816 leader_election.cc:302] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:50.732580 23194 raft_consensus.cc:2726] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:50.746270 9692 consensus_peers.cc:485] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:50.766683 6816 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:50.772487 6816 consensus_peers.cc:485] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:50.779152 7080 consensus_peers.cc:485] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:00:50.781457 23217 raft_consensus.cc:482] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:50.781599 23217 raft_consensus.cc:504] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:50.782079 23217 leader_election.cc:288] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:50.783465 9690 leader_election.cc:334] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:50.784355 9692 leader_election.cc:334] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:50.784438 9692 leader_election.cc:302] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:50.785495 23217 raft_consensus.cc:2726] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:50.790944 6947 consensus_peers.cc:485] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:50.807708 6818 consensus_peers.cc:485] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:50.829530 6949 consensus_peers.cc:485] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:50.873538 6947 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:00:50.883692 23209 raft_consensus.cc:482] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:50.883824 23209 raft_consensus.cc:504] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:50.884402 23209 leader_election.cc:288] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:50.886029 7080 leader_election.cc:334] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:50.886191 7080 leader_election.cc:334] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:50.886248 7080 leader_election.cc:302] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:50.886584 23209 raft_consensus.cc:2726] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:50.931648 6818 consensus_peers.cc:485] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:50.937498 9690 consensus_peers.cc:485] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:50.956437 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:50.981400 7078 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.000183 6947 consensus_peers.cc:485] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.031632 7209 consensus_peers.cc:485] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.048096 7080 consensus_peers.cc:485] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:51.089044 7080 consensus_peers.cc:485] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.105315 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.108582 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:51.111582 9692 consensus_peers.cc:485] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.126708 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:00:51.128546 23209 raft_consensus.cc:482] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:51.128671 23209 raft_consensus.cc:504] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:51.129063 23209 leader_election.cc:288] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:51.130048 7080 leader_election.cc:334] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:51.130234 7080 leader_election.cc:334] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:51.130295 7080 leader_election.cc:302] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:51.130446 23209 raft_consensus.cc:2726] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:51.131464 7211 consensus_peers.cc:485] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.140081 6949 consensus_peers.cc:485] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.145781 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.152235 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:51.176187 9692 consensus_peers.cc:485] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.187922 9692 consensus_peers.cc:485] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:51.225401 6947 consensus_peers.cc:485] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.241202 7209 consensus_peers.cc:485] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.244390 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.256145 6818 consensus_peers.cc:485] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.258421 6949 consensus_peers.cc:485] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.259179 7078 consensus_peers.cc:485] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.273218 6947 consensus_peers.cc:485] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.296352 6949 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.354314 7211 consensus_peers.cc:485] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.357753 6949 consensus_peers.cc:485] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.362910 6947 consensus_peers.cc:485] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.403348 7211 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.405925 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:51.412494 6816 consensus_peers.cc:485] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.431425 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.468737 6818 consensus_peers.cc:485] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.494501 9692 consensus_peers.cc:485] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.496500 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.506671 9692 consensus_peers.cc:485] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.518448 9690 consensus_peers.cc:485] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.518800 7211 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.573729 6818 consensus_peers.cc:485] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.580240 6818 consensus_peers.cc:485] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.590001 7078 consensus_peers.cc:485] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.592927 6947 consensus_peers.cc:485] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.605998 7211 consensus_peers.cc:485] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.621408 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:51.648564 7078 consensus_peers.cc:485] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.651978 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:51.656831 6818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111) [suppressed 261 similar messages]
W20240503 02:00:51.675407 6949 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111) [suppressed 256 similar messages]
W20240503 02:00:51.709029 7080 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111) [suppressed 217 similar messages]
W20240503 02:00:51.722998 9692 consensus_peers.cc:485] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.737171 7211 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111) [suppressed 196 similar messages]
W20240503 02:00:51.737937 7211 consensus_peers.cc:485] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.745738 7080 consensus_peers.cc:485] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.759982 7209 consensus_peers.cc:485] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.768209 9690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111) [suppressed 173 similar messages]
W20240503 02:00:51.769454 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.770895 7209 consensus_peers.cc:485] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.770972 7078 consensus_peers.cc:485] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.798537 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.866130 7211 consensus_peers.cc:485] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:51.870935 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:51.906306 6818 consensus_peers.cc:485] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.916312 7209 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.948979 9692 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:51.972322 6818 consensus_peers.cc:485] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:52.061794 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:52.096923 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:52.120173 7080 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:52.172339 7078 consensus_peers.cc:485] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:52.188907 7211 consensus_peers.cc:485] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:52.193535 9690 consensus_peers.cc:485] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:52.221741 9692 consensus_peers.cc:485] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:52.268707 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:52.271486 6818 consensus_peers.cc:485] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:52.272526 6818 consensus_peers.cc:485] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:52.313562 6949 consensus_peers.cc:485] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:52.338397 9690 consensus_peers.cc:485] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:00:52.364074 23209 raft_consensus.cc:482] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:52.364213 23209 raft_consensus.cc:504] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:52.364645 23209 leader_election.cc:288] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:52.365603 7078 leader_election.cc:334] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:52.365768 7080 leader_election.cc:334] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:52.365857 7080 leader_election.cc:302] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:52.365991 23209 raft_consensus.cc:2726] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:00:52.381865 23211 raft_consensus.cc:482] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:52.382025 23211 raft_consensus.cc:504] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:52.382453 23211 leader_election.cc:288] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:52.383996 9692 leader_election.cc:334] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:52.384127 9692 leader_election.cc:334] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:52.384191 9692 leader_election.cc:302] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:52.384317 23211 raft_consensus.cc:2726] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:52.424399 6818 consensus_peers.cc:485] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:52.449524 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:52.456498 9692 consensus_peers.cc:485] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:52.458412 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:52.493867 7078 consensus_peers.cc:485] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:52.514689 6949 consensus_peers.cc:485] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:52.550583 6818 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:52.558265 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:00:52.579550 7209 consensus_peers.cc:485] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:52.586886 9690 consensus_peers.cc:485] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:52.588815 6816 consensus_peers.cc:485] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:52.602190 9692 consensus_peers.cc:485] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:52.612407 7080 consensus_peers.cc:485] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:52.627990 7211 consensus_peers.cc:485] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:52.651340 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:52.688879 7211 consensus_peers.cc:485] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:00:52.689373 23220 raft_consensus.cc:482] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:52.689507 23220 raft_consensus.cc:504] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:00:52.690001 23220 leader_election.cc:288] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:52.691256 6947 leader_election.cc:334] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:52.691500 6949 leader_election.cc:334] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:52.691578 6949 leader_election.cc:302] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:52.691851 23220 raft_consensus.cc:2726] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:52.779748 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:52.835675 6949 consensus_peers.cc:485] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:52.854353 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:52.988349 7078 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.002485 7080 consensus_peers.cc:485] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.015807 6949 consensus_peers.cc:485] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:53.016886 6816 consensus_peers.cc:485] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:53.029544 7209 consensus_peers.cc:485] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:53.037804 6818 consensus_peers.cc:485] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.037827 6949 consensus_peers.cc:485] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:53.041888 7078 consensus_peers.cc:485] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.042496 6818 consensus_peers.cc:485] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.059772 6818 consensus_peers.cc:485] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.063794 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:53.090857 7211 consensus_peers.cc:485] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:53.091125 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:53.092222 6818 consensus_peers.cc:485] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:53.188081 6816 consensus_peers.cc:485] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.190809 7209 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 02:00:53.210801 23211 raft_consensus.cc:482] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:53.210968 23211 raft_consensus.cc:504] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:53.211426 23211 leader_election.cc:288] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:53.212757 9692 leader_election.cc:334] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:53.212869 9692 leader_election.cc:334] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:53.212922 9692 leader_election.cc:302] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:53.213059 23211 raft_consensus.cc:2726] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:53.222584 9692 consensus_peers.cc:485] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:53.224326 7080 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.228719 7080 consensus_peers.cc:485] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.283025 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:53.296331 7080 consensus_peers.cc:485] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.315104 6818 consensus_peers.cc:485] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.355194 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.376559 9690 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.380312 6947 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.384857 6949 consensus_peers.cc:485] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.398123 7078 consensus_peers.cc:485] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.422794 9692 consensus_peers.cc:485] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.449317 7211 consensus_peers.cc:485] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.456743 7078 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.456808 6947 consensus_peers.cc:485] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.471501 7209 consensus_peers.cc:485] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.482177 6818 consensus_peers.cc:485] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.502678 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:53.528739 6816 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.598570 9692 consensus_peers.cc:485] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:53.609537 9690 consensus_peers.cc:485] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.610752 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:53.612275 9692 consensus_peers.cc:485] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.618901 6947 consensus_peers.cc:485] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.633438 7080 consensus_peers.cc:485] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:53.648118 6949 consensus_peers.cc:485] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.682090 6949 consensus_peers.cc:485] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.698570 6947 consensus_peers.cc:485] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.731144 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:53.748706 6949 consensus_peers.cc:485] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.759455 6818 consensus_peers.cc:485] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.767344 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.776115 7078 consensus_peers.cc:485] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.783293 6949 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.785502 6818 consensus_peers.cc:485] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.792385 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.826139 6947 consensus_peers.cc:485] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.827046 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.829069 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.864470 7078 consensus_peers.cc:485] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.933651 6816 consensus_peers.cc:485] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.950834 7078 consensus_peers.cc:485] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.953491 9692 consensus_peers.cc:485] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:53.962894 7209 consensus_peers.cc:485] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.002142 9692 consensus_peers.cc:485] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.008594 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:54.047935 6947 consensus_peers.cc:485] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.051162 6818 consensus_peers.cc:485] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.051192 7211 consensus_peers.cc:485] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.059350 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.075523 7209 consensus_peers.cc:485] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.078145 9690 consensus_peers.cc:485] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.093526 7211 consensus_peers.cc:485] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.110724 6947 consensus_peers.cc:485] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.111968 7211 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.112536 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:54.130649 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.136188 7211 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.243894 7209 consensus_peers.cc:485] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.243927 9692 consensus_peers.cc:485] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.267648 6818 consensus_peers.cc:485] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.272631 7211 consensus_peers.cc:485] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.291682 7078 consensus_peers.cc:485] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.309216 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:54.331725 7080 consensus_peers.cc:485] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.338411 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.365501 7211 consensus_peers.cc:485] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240503 02:00:54.388599 23142 raft_consensus.cc:482] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:54.388741 23142 raft_consensus.cc:504] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:54.389194 23142 leader_election.cc:288] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:54.390344 7080 leader_election.cc:334] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:54.390436 7078 leader_election.cc:334] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:54.390545 7078 leader_election.cc:302] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:54.390789 23142 raft_consensus.cc:2726] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:54.442400 6949 consensus_peers.cc:485] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.490317 9692 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.495519 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:54.512861 6818 consensus_peers.cc:485] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.515805 7209 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.614427 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:54.648013 7211 consensus_peers.cc:485] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.714552 6818 consensus_peers.cc:485] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.717307 6818 consensus_peers.cc:485] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.718327 7078 consensus_peers.cc:485] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.719252 6818 consensus_peers.cc:485] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.721967 9690 consensus_peers.cc:485] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.788326 7080 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.789402 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:54.795152 9692 consensus_peers.cc:485] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.800576 6818 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.822060 9690 consensus_peers.cc:485] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.864989 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:54.874826 7080 consensus_peers.cc:485] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.881731 6818 consensus_peers.cc:485] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.941494 6949 consensus_peers.cc:485] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:54.990839 6816 consensus_peers.cc:485] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:54.993099 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:55.009714 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:55.041687 7211 consensus_peers.cc:485] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:55.048079 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:00:55.053462 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:55.067205 7078 consensus_peers.cc:485] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:55.095681 7211 consensus_peers.cc:485] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:55.103150 9692 consensus_peers.cc:485] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:55.156451 9690 consensus_peers.cc:485] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:55.182776 9692 consensus_peers.cc:485] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:55.265270 7209 consensus_peers.cc:485] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:55.291827 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:55.297508 23217 raft_consensus.cc:482] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:55.297632 23217 raft_consensus.cc:504] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:00:55.298192 23217 leader_election.cc:288] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:55.299389 9690 leader_election.cc:334] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:55.299680 9692 leader_election.cc:334] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:55.299757 9692 leader_election.cc:302] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:55.300055 23217 raft_consensus.cc:2726] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:55.300524 7080 consensus_peers.cc:485] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:55.344487 6949 consensus_peers.cc:485] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:55.363608 7078 consensus_peers.cc:485] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:55.378950 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:55.409394 6818 consensus_peers.cc:485] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:55.412322 6949 consensus_peers.cc:485] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:55.562885 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:55.571081 7078 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:55.575085 6816 consensus_peers.cc:485] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:55.597453 6818 consensus_peers.cc:485] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:55.602031 6818 consensus_peers.cc:485] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:55.607014 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:55.614511 7209 consensus_peers.cc:485] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:55.636292 9692 consensus_peers.cc:485] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:55.680624 7211 consensus_peers.cc:485] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:55.708768 7080 consensus_peers.cc:485] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:55.716115 7211 consensus_peers.cc:485] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:55.768941 6949 consensus_peers.cc:485] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:55.790289 6816 consensus_peers.cc:485] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:55.793874 7080 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:55.848145 7209 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:55.865249 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:55.873078 6818 consensus_peers.cc:485] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:55.886108 9690 consensus_peers.cc:485] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:55.892815 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:55.910112 7080 consensus_peers.cc:485] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:55.938625 6947 consensus_peers.cc:485] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:55.948417 6947 consensus_peers.cc:485] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:55.963053 7078 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:55.971051 6947 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:55.977013 6816 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:55.993530 6818 consensus_peers.cc:485] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.027642 6949 consensus_peers.cc:485] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.029429 6947 consensus_peers.cc:485] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.034562 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:56.062271 9690 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.064873 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.068245 6818 consensus_peers.cc:485] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.069171 9692 consensus_peers.cc:485] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:56.086755 7080 consensus_peers.cc:485] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:56.131716 7078 consensus_peers.cc:485] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.136883 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:56.150383 6947 consensus_peers.cc:485] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.172225 9692 consensus_peers.cc:485] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.229339 9692 consensus_peers.cc:485] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.231647 6949 consensus_peers.cc:485] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.256490 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.276075 6818 consensus_peers.cc:485] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.291785 6949 consensus_peers.cc:485] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.306355 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:56.316843 7209 consensus_peers.cc:485] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.323643 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.334812 6818 consensus_peers.cc:485] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.347890 7211 consensus_peers.cc:485] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.362265 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.365276 6949 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.384550 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.391041 6949 consensus_peers.cc:485] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.397430 7078 consensus_peers.cc:485] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.409664 7209 consensus_peers.cc:485] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.470930 7078 consensus_peers.cc:485] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.493345 6818 consensus_peers.cc:485] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.517347 6816 consensus_peers.cc:485] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.518136 9692 consensus_peers.cc:485] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.523316 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:56.531787 7209 consensus_peers.cc:485] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.538458 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.548199 9690 consensus_peers.cc:485] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.561646 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:56.574348 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.581529 7078 consensus_peers.cc:485] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.607825 7211 consensus_peers.cc:485] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.609601 6947 consensus_peers.cc:485] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.609838 9692 consensus_peers.cc:485] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.627271 6818 consensus_peers.cc:485] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.638731 6947 consensus_peers.cc:485] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.672616 7209 consensus_peers.cc:485] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.701689 6818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111) [suppressed 252 similar messages]
W20240503 02:00:56.702354 6818 consensus_peers.cc:485] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.723806 7078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111) [suppressed 204 similar messages]
W20240503 02:00:56.724725 7078 consensus_peers.cc:485] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.725064 9692 consensus_peers.cc:485] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.737648 7209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111) [suppressed 196 similar messages]
W20240503 02:00:56.746502 6949 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111) [suppressed 255 similar messages]
W20240503 02:00:56.764495 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:56.771490 7211 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:00:56.784968 23228 raft_consensus.cc:482] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:56.785109 23228 raft_consensus.cc:504] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:00:56.785542 23228 leader_election.cc:288] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:56.788467 7209 leader_election.cc:334] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:56.789146 7211 leader_election.cc:334] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:56.789250 7211 leader_election.cc:302] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:56.789419 23228 raft_consensus.cc:2726] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:56.798815 7211 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.817425 9692 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111) [suppressed 176 similar messages]
W20240503 02:00:56.876855 7211 consensus_peers.cc:485] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.877494 7211 consensus_peers.cc:485] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:56.927088 6818 consensus_peers.cc:485] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.944665 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:56.962560 9692 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.965443 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:56.995276 9690 consensus_peers.cc:485] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:56.999359 7080 consensus_peers.cc:485] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:57.003190 6949 consensus_peers.cc:485] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:57.029762 6818 consensus_peers.cc:485] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:57.099656 7209 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:57.105269 9692 consensus_peers.cc:485] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:57.109376 7080 consensus_peers.cc:485] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:57.118723 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:57.130169 7080 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 02:00:57.164217 23250 raft_consensus.cc:482] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:57.164454 23250 raft_consensus.cc:504] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:00:57.164925 23250 leader_election.cc:288] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:00:57.166353 9690 leader_election.cc:334] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:57.167348 9692 leader_election.cc:334] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:57.167472 9692 leader_election.cc:302] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:57.167651 23250 raft_consensus.cc:2726] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:57.177269 7211 consensus_peers.cc:485] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:57.243314 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:57.265641 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:57.282428 6818 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:57.283211 6949 consensus_peers.cc:485] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:57.348443 6816 consensus_peers.cc:485] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:57.361014 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:57.361104 6818 consensus_peers.cc:485] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:57.368845 7078 consensus_peers.cc:485] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:57.386834 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:57.402580 9690 consensus_peers.cc:485] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:57.429761 7211 consensus_peers.cc:485] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:57.454449 7211 consensus_peers.cc:485] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:57.468093 9692 consensus_peers.cc:485] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:57.515611 9690 consensus_peers.cc:485] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:57.546185 6818 consensus_peers.cc:485] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:57.588299 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:00:57.616180 7078 consensus_peers.cc:485] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:57.689838 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:57.747026 7078 consensus_peers.cc:485] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:57.766256 7209 consensus_peers.cc:485] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:57.784996 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:57.812005 6949 consensus_peers.cc:485] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:57.843900 7078 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:57.865403 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:57.871724 7080 consensus_peers.cc:485] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:57.909705 9692 consensus_peers.cc:485] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:57.972849 7211 consensus_peers.cc:485] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.000586 6949 consensus_peers.cc:485] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:58.002420 6818 consensus_peers.cc:485] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.025779 6818 consensus_peers.cc:485] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.045818 6949 consensus_peers.cc:485] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:58.074913 6818 consensus_peers.cc:485] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.141045 7211 consensus_peers.cc:485] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:58.186561 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.187731 9692 consensus_peers.cc:485] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:58.236564 7209 consensus_peers.cc:485] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:58.266819 6816 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.272574 7080 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.309906 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:58.319350 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:58.319579 7080 consensus_peers.cc:485] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.327886 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.352342 6816 consensus_peers.cc:485] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.354877 9690 consensus_peers.cc:485] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.361435 7078 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.361737 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:58.407472 9692 consensus_peers.cc:485] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.410461 7080 consensus_peers.cc:485] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.416500 6947 consensus_peers.cc:485] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.418391 7209 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.423635 7209 consensus_peers.cc:485] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.443358 6818 consensus_peers.cc:485] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:58.448534 6816 consensus_peers.cc:485] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:58.482421 7078 consensus_peers.cc:485] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.483973 6947 consensus_peers.cc:485] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.509649 6949 consensus_peers.cc:485] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.512187 9692 consensus_peers.cc:485] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:58.547462 6949 consensus_peers.cc:485] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.581485 7080 consensus_peers.cc:485] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:58.585201 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.596473 6947 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.619150 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.622853 9690 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.636355 7211 consensus_peers.cc:485] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.655290 6818 consensus_peers.cc:485] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.655732 6947 consensus_peers.cc:485] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 02:00:58.656554 23228 raft_consensus.cc:482] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:00:58.656663 23228 raft_consensus.cc:504] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:58.657053 23228 leader_election.cc:288] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:58.658396 7211 leader_election.cc:334] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:58.658555 7211 leader_election.cc:334] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:58.658607 7211 leader_election.cc:302] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:00:58.658771 23228 raft_consensus.cc:2726] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:58.672410 9692 consensus_peers.cc:485] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.697438 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.702605 6818 consensus_peers.cc:485] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.706069 6818 consensus_peers.cc:485] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 02:00:58.763197 23169 raft_consensus.cc:482] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:00:58.763382 23169 raft_consensus.cc:504] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:00:58.764159 23169 leader_election.cc:288] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:58.765645 6818 leader_election.cc:334] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:58.766131 6816 leader_election.cc:334] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:58.766227 6816 leader_election.cc:302] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:58.766475 23169 raft_consensus.cc:2726] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:58.771242 7209 consensus_peers.cc:485] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.789669 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:58.801496 6949 consensus_peers.cc:485] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.843878 6818 consensus_peers.cc:485] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.859577 6818 consensus_peers.cc:485] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.859752 6949 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.863970 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:58.869957 6947 consensus_peers.cc:485] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.905129 9690 consensus_peers.cc:485] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.917686 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:58.923225 6949 consensus_peers.cc:485] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.962157 7209 consensus_peers.cc:485] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.962404 9692 consensus_peers.cc:485] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.983158 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:58.998134 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.018426 7078 consensus_peers.cc:485] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.027480 6947 consensus_peers.cc:485] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.045342 7211 consensus_peers.cc:485] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.111321 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.115265 7078 consensus_peers.cc:485] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.117348 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.130409 7209 consensus_peers.cc:485] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.156265 6818 consensus_peers.cc:485] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.167661 7211 consensus_peers.cc:485] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 02:00:59.177107 23249 raft_consensus.cc:482] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:59.177254 23249 raft_consensus.cc:504] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:00:59.177714 23249 leader_election.cc:288] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:00:59.178894 6947 leader_election.cc:334] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:59.179781 6949 leader_election.cc:334] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:00:59.179852 6949 leader_election.cc:302] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:59.179980 23249 raft_consensus.cc:2726] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:59.187852 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:59.189541 6818 consensus_peers.cc:485] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.197796 7078 consensus_peers.cc:485] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.204938 9692 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.206780 6947 consensus_peers.cc:485] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.219633 7078 consensus_peers.cc:485] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.223256 7211 consensus_peers.cc:485] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.262436 9692 consensus_peers.cc:485] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.263317 7211 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.287480 9692 consensus_peers.cc:485] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.315413 6816 consensus_peers.cc:485] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.319170 7211 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.339393 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:59.349944 6949 consensus_peers.cc:485] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.368749 7211 consensus_peers.cc:485] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:59.391206 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:00:59.422726 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:59.455235 7080 consensus_peers.cc:485] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.455428 7209 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.532325 7080 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.533448 9692 consensus_peers.cc:485] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.559679 7080 consensus_peers.cc:485] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.640287 6818 consensus_peers.cc:485] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.657120 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:00:59.727643 9690 consensus_peers.cc:485] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.741752 6949 consensus_peers.cc:485] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.749941 6818 consensus_peers.cc:485] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.770427 9690 consensus_peers.cc:485] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.784812 6818 consensus_peers.cc:485] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.816555 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:00:59.833614 23258 raft_consensus.cc:482] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:00:59.833770 23258 raft_consensus.cc:504] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:00:59.834264 23258 leader_election.cc:288] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:00:59.835698 7080 leader_election.cc:334] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:00:59.835844 7078 leader_election.cc:334] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:59.835917 7078 leader_election.cc:302] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:00:59.836046 23258 raft_consensus.cc:2726] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:00:59.852931 7211 consensus_peers.cc:485] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.854413 9690 consensus_peers.cc:485] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.906188 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:00:59.913636 6816 consensus_peers.cc:485] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:00:59.927017 6818 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.963981 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.966034 9692 consensus_peers.cc:485] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:00:59.976822 7078 consensus_peers.cc:485] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:00.008853 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:01:00.023733 7211 consensus_peers.cc:485] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:00.043447 6818 consensus_peers.cc:485] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:00.069639 6949 consensus_peers.cc:485] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:00.248693 7078 consensus_peers.cc:485] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:00.269126 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:00.313444 9692 consensus_peers.cc:485] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:00.345074 7209 consensus_peers.cc:485] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:00.360662 7080 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:00.368149 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:00.374461 6818 consensus_peers.cc:485] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:00.409406 7080 consensus_peers.cc:485] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:00.414662 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:00.430685 7078 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:00.451224 7211 consensus_peers.cc:485] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:00.455689 6949 consensus_peers.cc:485] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:00.466315 7211 consensus_peers.cc:485] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:00.494490 7080 consensus_peers.cc:485] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:00.501574 6949 consensus_peers.cc:485] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:00.529258 7078 consensus_peers.cc:485] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:00.570634 6818 consensus_peers.cc:485] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:01:00.577739 23266 raft_consensus.cc:482] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:01:00.577894 23266 raft_consensus.cc:504] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:01:00.578236 23266 leader_election.cc:288] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:01:00.579668 6949 leader_election.cc:334] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:00.579852 6949 leader_election.cc:334] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:00.579916 6949 leader_election.cc:302] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:00.580072 23266 raft_consensus.cc:2726] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:01:00.593997 23258 raft_consensus.cc:482] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:01:00.594210 23258 raft_consensus.cc:504] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:01:00.594842 23258 leader_election.cc:288] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:01:00.596326 7080 leader_election.cc:334] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:00.596457 7078 leader_election.cc:334] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:01:00.596521 7078 leader_election.cc:302] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:00.596673 23258 raft_consensus.cc:2726] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:00.611691 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:00.615494 6818 consensus_peers.cc:485] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:00.615867 7080 consensus_peers.cc:485] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:00.655165 7209 consensus_peers.cc:485] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:01:00.671329 23260 raft_consensus.cc:482] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:01:00.671505 23260 raft_consensus.cc:504] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:01:00.671972 23260 leader_election.cc:288] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:01:00.672979 9690 leader_election.cc:334] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:00.673252 9692 leader_election.cc:334] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:00.673367 9692 leader_election.cc:302] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:00.673568 23260 raft_consensus.cc:2726] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:00.736426 6816 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:00.748291 9692 consensus_peers.cc:485] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:00.784049 9692 consensus_peers.cc:485] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:00.790982 7078 consensus_peers.cc:485] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:00.810876 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:00.832748 6947 consensus_peers.cc:485] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:00.864923 9690 consensus_peers.cc:485] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:00.878592 7078 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:00.881395 6816 consensus_peers.cc:485] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:00.886298 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:00.888586 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:00.895285 7211 consensus_peers.cc:485] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:00.931532 6949 consensus_peers.cc:485] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:00.951155 7080 consensus_peers.cc:485] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:00.958127 6818 consensus_peers.cc:485] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:00.958618 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:00.970260 7209 consensus_peers.cc:485] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:00.986804 9692 consensus_peers.cc:485] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:00.992106 9690 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.008423 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:01.018307 6949 consensus_peers.cc:485] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.045128 6947 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.061712 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.076159 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.082415 6816 consensus_peers.cc:485] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.107035 6949 consensus_peers.cc:485] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.119294 6949 consensus_peers.cc:485] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:01:01.119576 23211 raft_consensus.cc:482] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:01:01.119676 23211 raft_consensus.cc:504] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:01:01.120184 23211 leader_election.cc:288] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943)
I20240503 02:01:01.120554 6882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49eabbb8972840759ffa83951cb0b3b7" candidate_uuid: "db1a0000aaa346f79c0aa4b6c5105191" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" is_pre_election: true
I20240503 02:01:01.120700 6882 raft_consensus.cc:2454] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1a0000aaa346f79c0aa4b6c5105191 in term 1.
W20240503 02:01:01.121084 9690 leader_election.cc:334] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:01.121685 9692 leader_election.cc:334] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:01:01.121757 9692 leader_election.cc:302] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, db1a0000aaa346f79c0aa4b6c5105191; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:01.121977 23211 raft_consensus.cc:2726] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:01.139874 6818 consensus_peers.cc:485] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.184626 7209 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.194535 6818 consensus_peers.cc:485] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:01.214355 6947 consensus_peers.cc:485] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.219506 9692 consensus_peers.cc:485] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:01:01.234670 23211 raft_consensus.cc:482] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:01:01.234812 23211 raft_consensus.cc:504] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:01:01.235193 23211 leader_election.cc:288] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:01:01.236330 9692 leader_election.cc:334] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:01.236472 9692 leader_election.cc:334] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:01.236518 9692 leader_election.cc:302] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:01.236682 23211 raft_consensus.cc:2726] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:01.294824 6818 consensus_peers.cc:485] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.322877 7209 consensus_peers.cc:485] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.365569 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:01.368351 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.374680 7078 consensus_peers.cc:485] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.404090 7078 consensus_peers.cc:485] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.413396 9692 consensus_peers.cc:485] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.438709 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:01.444371 6947 consensus_peers.cc:485] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:01:01.463546 23246 raft_consensus.cc:482] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:01:01.463701 23246 raft_consensus.cc:504] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:01:01.464082 23246 leader_election.cc:288] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:01:01.465080 6947 consensus_peers.cc:485] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.465183 6947 leader_election.cc:334] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:01.465432 6949 leader_election.cc:334] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:01:01.465502 6949 leader_election.cc:302] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:01.465638 23246 raft_consensus.cc:2726] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:01.501477 7209 consensus_peers.cc:485] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.501621 7078 consensus_peers.cc:485] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.530628 6818 consensus_peers.cc:485] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.555128 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:01.597558 7209 consensus_peers.cc:485] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.603685 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.617625 6947 consensus_peers.cc:485] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.620663 6818 consensus_peers.cc:485] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.627702 6949 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.640497 9690 consensus_peers.cc:485] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.651901 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.662665 7211 consensus_peers.cc:485] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.669768 9692 consensus_peers.cc:485] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.682631 6818 consensus_peers.cc:485] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.709941 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.715373 9692 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.717198 7211 consensus_peers.cc:485] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.730324 6818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111) [suppressed 249 similar messages]
W20240503 02:01:01.731074 6818 consensus_peers.cc:485] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.764142 7211 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111) [suppressed 205 similar messages]
W20240503 02:01:01.764973 7211 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.775579 6949 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111) [suppressed 258 similar messages]
W20240503 02:01:01.777117 7078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111) [suppressed 206 similar messages]
W20240503 02:01:01.777983 7078 consensus_peers.cc:485] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.801435 6949 consensus_peers.cc:485] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.816215 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:01.833468 9690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111) [suppressed 179 similar messages]
W20240503 02:01:01.848242 6816 consensus_peers.cc:485] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.848390 7211 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.864231 9692 consensus_peers.cc:485] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.868945 7080 consensus_peers.cc:485] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.876550 7211 consensus_peers.cc:485] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:01.897629 7080 consensus_peers.cc:485] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.940438 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:01.949935 7211 consensus_peers.cc:485] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:01:01.953395 23275 raft_consensus.cc:482] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:01:01.953635 23275 raft_consensus.cc:504] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:01:01.954102 23275 leader_election.cc:288] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:01:01.955564 7080 leader_election.cc:334] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:01.955718 7080 leader_election.cc:334] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:01:01.955775 7080 leader_election.cc:302] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:01.955926 23275 raft_consensus.cc:2726] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:01.957425 9692 consensus_peers.cc:485] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.958678 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:01.985934 7209 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:01.993921 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:02.008286 6947 consensus_peers.cc:485] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:02.109259 6818 consensus_peers.cc:485] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:02.112085 7080 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:02.142455 6816 consensus_peers.cc:485] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:02.158689 6818 consensus_peers.cc:485] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:02.163298 9690 consensus_peers.cc:485] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:02.184224 6818 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:02.243095 7211 consensus_peers.cc:485] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:02.255184 6949 consensus_peers.cc:485] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:02.299871 9690 consensus_peers.cc:485] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:02.311622 9690 consensus_peers.cc:485] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:02.341492 7078 consensus_peers.cc:485] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:01:02.342231 23169 raft_consensus.cc:482] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:01:02.342348 23169 raft_consensus.cc:504] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:01:02.342849 23169 leader_election.cc:288] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:01:02.344345 6816 leader_election.cc:334] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:02.344745 6818 leader_election.cc:334] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:01:02.344843 6818 leader_election.cc:302] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:02.344998 23169 raft_consensus.cc:2726] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:02.363931 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:02.365396 9692 consensus_peers.cc:485] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:02.403995 6949 consensus_peers.cc:485] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:02.427217 6818 consensus_peers.cc:485] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:01:02.432515 23279 raft_consensus.cc:482] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:01:02.432680 23279 raft_consensus.cc:504] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:01:02.433146 23279 leader_election.cc:288] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:01:02.434625 9692 leader_election.cc:334] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:02.434772 9692 leader_election.cc:334] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:01:02.434846 9692 leader_election.cc:302] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:02.435019 23279 raft_consensus.cc:2726] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:02.489537 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:02.490427 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:01:02.501492 6818 consensus_peers.cc:485] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:02.571875 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:02.637712 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:02.664086 7211 consensus_peers.cc:485] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:02.754841 7078 consensus_peers.cc:485] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:02.762395 6818 consensus_peers.cc:485] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:02.793252 9692 consensus_peers.cc:485] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:02.821146 6949 consensus_peers.cc:485] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:02.853792 6949 consensus_peers.cc:485] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:02.861936 7211 consensus_peers.cc:485] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:02.876406 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:02.876435 7209 consensus_peers.cc:485] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:02.903009 7080 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:02.907423 7211 consensus_peers.cc:485] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:02.952014 7078 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:02.952214 7080 consensus_peers.cc:485] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 02:01:02.980144 23282 raft_consensus.cc:482] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:01:02.980391 23282 raft_consensus.cc:504] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:01:02.980921 23282 leader_election.cc:288] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:01:02.982666 6949 leader_election.cc:334] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:02.982838 6949 leader_election.cc:334] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:02.982934 6949 leader_election.cc:302] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:02.983139 23282 raft_consensus.cc:2726] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:02.991433 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:03.024818 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.033324 7080 consensus_peers.cc:485] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.048784 7078 consensus_peers.cc:485] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.119978 6818 consensus_peers.cc:485] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.120858 9692 consensus_peers.cc:485] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:03.144749 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:03.183023 7078 consensus_peers.cc:485] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.223943 7078 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.256462 7080 consensus_peers.cc:485] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.264190 7209 consensus_peers.cc:485] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.292049 7211 consensus_peers.cc:485] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.307317 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.322206 6818 consensus_peers.cc:485] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.329344 9690 consensus_peers.cc:485] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.354097 6949 consensus_peers.cc:485] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.385867 6947 consensus_peers.cc:485] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.398073 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:03.423586 6949 consensus_peers.cc:485] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.427672 6816 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.431300 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.453186 7209 consensus_peers.cc:485] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:03.461803 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.488600 9692 consensus_peers.cc:485] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:03.492619 9692 consensus_peers.cc:485] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.518640 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:01:03.525270 23274 raft_consensus.cc:482] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:01:03.525408 23274 raft_consensus.cc:504] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:01:03.525857 23274 leader_election.cc:288] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:01:03.527159 7078 leader_election.cc:334] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:03.527866 7080 leader_election.cc:334] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:03.527932 7080 leader_election.cc:302] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:03.528069 23274 raft_consensus.cc:2726] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:03.532781 6816 consensus_peers.cc:485] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:03.561972 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.605816 6947 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.606316 6818 consensus_peers.cc:485] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.629740 9690 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.630699 6949 consensus_peers.cc:485] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.641024 7209 consensus_peers.cc:485] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.643762 6818 consensus_peers.cc:485] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.660647 6818 consensus_peers.cc:485] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.665699 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:03.672230 23274 raft_consensus.cc:482] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:01:03.672391 23274 raft_consensus.cc:504] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:01:03.672859 23274 leader_election.cc:288] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:01:03.673972 7078 leader_election.cc:334] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:03.674266 7080 leader_election.cc:334] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:01:03.674350 7080 leader_election.cc:302] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
W20240503 02:01:03.674476 7080 consensus_peers.cc:485] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:01:03.674547 23274 raft_consensus.cc:2726] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:03.674600 7080 leader_election.cc:334] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:03.705369 6816 consensus_peers.cc:485] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.741070 7078 consensus_peers.cc:485] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.750620 6949 consensus_peers.cc:485] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 02:01:03.758821 23169 raft_consensus.cc:482] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:01:03.759003 23169 raft_consensus.cc:504] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:01:03.759692 23169 leader_election.cc:288] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:01:03.760561 6816 leader_election.cc:334] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:03.760661 6818 leader_election.cc:334] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:03.760753 6818 leader_election.cc:302] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:03.760910 23169 raft_consensus.cc:2726] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:03.766700 6947 consensus_peers.cc:485] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.777228 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:03.881002 9692 consensus_peers.cc:485] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.889873 6947 consensus_peers.cc:485] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.890383 7209 consensus_peers.cc:485] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.896741 7209 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.904515 9690 consensus_peers.cc:485] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.910001 6818 consensus_peers.cc:485] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.934042 7078 consensus_peers.cc:485] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.961872 6949 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:03.976075 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.004120 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.012279 6947 consensus_peers.cc:485] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.013013 6818 consensus_peers.cc:485] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.021135 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.023641 6818 consensus_peers.cc:485] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:04.051568 6947 consensus_peers.cc:485] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.056557 7209 consensus_peers.cc:485] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.074435 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:04.159005 7211 consensus_peers.cc:485] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.174407 7211 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 02:01:04.197768 23276 raft_consensus.cc:482] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:01:04.197929 23276 raft_consensus.cc:504] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:01:04.198371 23276 leader_election.cc:288] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:01:04.199779 6949 leader_election.cc:334] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:04.199913 6949 leader_election.cc:334] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:04.199967 6949 leader_election.cc:302] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:04.200206 23294 raft_consensus.cc:2726] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:04.242465 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:04.247170 6818 consensus_peers.cc:485] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.248606 7078 consensus_peers.cc:485] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.250819 9692 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.269418 7211 consensus_peers.cc:485] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:04.279276 7078 consensus_peers.cc:485] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.279668 9692 consensus_peers.cc:485] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.281667 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:04.291049 9692 consensus_peers.cc:485] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.330076 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.343477 7080 consensus_peers.cc:485] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.347641 6947 consensus_peers.cc:485] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.360949 6949 consensus_peers.cc:485] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.362010 6818 consensus_peers.cc:485] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.381400 9692 consensus_peers.cc:485] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.381865 7211 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.433408 7211 consensus_peers.cc:485] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.446324 9690 consensus_peers.cc:485] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.448995 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:04.465510 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:04.494309 7080 consensus_peers.cc:485] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.573498 6818 consensus_peers.cc:485] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.579614 7211 consensus_peers.cc:485] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.604141 7080 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.605576 7209 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.638723 9692 consensus_peers.cc:485] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.638794 6816 consensus_peers.cc:485] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.656390 6818 consensus_peers.cc:485] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.690654 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:04.698961 6816 consensus_peers.cc:485] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:04.702983 6818 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.705881 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:04.711941 9692 consensus_peers.cc:485] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.758275 7211 consensus_peers.cc:485] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.760342 9690 consensus_peers.cc:485] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.782649 6818 consensus_peers.cc:485] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.865492 6949 consensus_peers.cc:485] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.883428 6949 consensus_peers.cc:485] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.904421 6818 consensus_peers.cc:485] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.906960 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:01:04.936232 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:04.945324 9690 consensus_peers.cc:485] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:04.978852 7078 consensus_peers.cc:485] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:05.094849 6818 consensus_peers.cc:485] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:05.109908 7078 consensus_peers.cc:485] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:05.142688 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:05.190773 7211 consensus_peers.cc:485] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240503 02:01:05.237788 23194 raft_consensus.cc:482] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:01:05.237962 23194 raft_consensus.cc:504] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:01:05.238607 23194 leader_election.cc:288] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:01:05.239059 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:05.239989 6816 leader_election.cc:334] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:05.240425 6818 leader_election.cc:334] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:05.240514 6818 leader_election.cc:302] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:05.240669 23194 raft_consensus.cc:2726] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:05.311576 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:05.319573 6949 consensus_peers.cc:485] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:05.360471 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:05.368595 7080 consensus_peers.cc:485] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:05.369047 6949 consensus_peers.cc:485] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:05.407126 7078 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:05.407171 9692 consensus_peers.cc:485] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:05.415709 7080 consensus_peers.cc:485] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:05.416868 7080 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:05.432891 7211 consensus_peers.cc:485] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:05.453270 6818 consensus_peers.cc:485] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:05.474447 9692 consensus_peers.cc:485] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:05.492563 7211 consensus_peers.cc:485] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:05.499915 7209 consensus_peers.cc:485] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:05.569077 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:05.618340 7209 consensus_peers.cc:485] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:05.642589 7078 consensus_peers.cc:485] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:05.649317 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:01:05.658272 23279 raft_consensus.cc:482] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:01:05.658397 23279 raft_consensus.cc:504] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:01:05.658869 23279 leader_election.cc:288] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:01:05.659911 9690 leader_election.cc:334] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:05.660001 9692 leader_election.cc:334] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:05.660061 9692 leader_election.cc:302] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:05.660194 23279 raft_consensus.cc:2726] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:05.679831 9692 consensus_peers.cc:485] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:05.688012 7078 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:05.716053 7080 consensus_peers.cc:485] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:05.724390 6818 consensus_peers.cc:485] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:05.805356 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:05.816056 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:05.829080 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:05.837393 6947 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:05.841173 7078 consensus_peers.cc:485] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:05.841812 6947 consensus_peers.cc:485] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:05.859985 6949 consensus_peers.cc:485] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:05.862164 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:05.867431 6816 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:05.923899 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:05.937150 9692 consensus_peers.cc:485] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:05.949954 9690 consensus_peers.cc:485] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:05.978169 6818 consensus_peers.cc:485] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.014288 7211 consensus_peers.cc:485] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.035240 6949 consensus_peers.cc:485] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.046837 7080 consensus_peers.cc:485] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:06.050694 7209 consensus_peers.cc:485] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:06.061224 6818 consensus_peers.cc:485] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.138176 6949 consensus_peers.cc:485] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.163758 6816 consensus_peers.cc:485] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.181422 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:06.183154 6818 consensus_peers.cc:485] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.185786 7078 consensus_peers.cc:485] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.230568 6949 consensus_peers.cc:485] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.286082 9690 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.292421 9690 consensus_peers.cc:485] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.296160 7078 consensus_peers.cc:485] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.303611 6818 consensus_peers.cc:485] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.326754 7209 consensus_peers.cc:485] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.333884 6816 consensus_peers.cc:485] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:06.347801 6947 consensus_peers.cc:485] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.406812 6947 consensus_peers.cc:485] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.409914 7209 consensus_peers.cc:485] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.413779 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:06.429436 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:06.455395 9692 consensus_peers.cc:485] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.476914 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.477123 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.490043 6818 consensus_peers.cc:485] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:06.498219 7209 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.501245 7078 consensus_peers.cc:485] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.568547 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.579073 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:06.581231 9692 consensus_peers.cc:485] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.590682 6818 consensus_peers.cc:485] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.591447 6947 consensus_peers.cc:485] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.594426 6949 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.609817 7211 consensus_peers.cc:485] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:01:06.627851 23279 raft_consensus.cc:482] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:01:06.628001 23279 raft_consensus.cc:504] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:01:06.628413 23279 leader_election.cc:288] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:01:06.629431 9690 leader_election.cc:334] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:06.629719 9692 leader_election.cc:334] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:06.629784 9692 leader_election.cc:302] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:06.629953 23279 raft_consensus.cc:2726] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:06.634083 7209 consensus_peers.cc:485] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.644151 6947 consensus_peers.cc:485] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.646042 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.658387 9692 consensus_peers.cc:485] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.736657 6818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111) [suppressed 253 similar messages]
I20240503 02:01:06.739668 23306 raft_consensus.cc:482] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:01:06.739785 23306 raft_consensus.cc:504] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:01:06.740109 23306 leader_election.cc:288] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:01:06.741308 6818 leader_election.cc:334] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:06.741410 6818 leader_election.cc:334] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:06.741441 6818 leader_election.cc:302] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:06.741676 23306 raft_consensus.cc:2726] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:06.758447 7080 consensus_peers.cc:485] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.763530 7211 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.771695 7078 consensus_peers.cc:485] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.772092 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.775136 9692 consensus_peers.cc:485] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.779352 7211 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111) [suppressed 200 similar messages]
W20240503 02:01:06.780246 7211 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.814391 7078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111) [suppressed 209 similar messages]
W20240503 02:01:06.832309 7080 consensus_peers.cc:485] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.850666 6947 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111) [suppressed 255 similar messages]
W20240503 02:01:06.851450 6947 consensus_peers.cc:485] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.873361 6818 consensus_peers.cc:485] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.887949 9692 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111) [suppressed 174 similar messages]
W20240503 02:01:06.905359 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:06.914286 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:06.938735 6816 consensus_peers.cc:485] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:06.941206 9690 consensus_peers.cc:485] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.949895 7211 consensus_peers.cc:485] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:06.952553 7211 consensus_peers.cc:485] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:06.975700 9692 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:07.010414 6949 consensus_peers.cc:485] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:07.026077 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:07.028093 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:07.066155 9692 consensus_peers.cc:485] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:07.067703 7209 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:07.083032 7080 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:07.097644 6949 consensus_peers.cc:485] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:07.105165 6818 consensus_peers.cc:485] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:07.112200 6818 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:07.126819 6816 consensus_peers.cc:485] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:07.130569 6818 consensus_peers.cc:485] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:07.148524 7211 consensus_peers.cc:485] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:07.202484 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:01:07.317934 6818 consensus_peers.cc:485] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:07.349076 6818 consensus_peers.cc:485] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:07.370357 6949 consensus_peers.cc:485] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:07.370796 7211 consensus_peers.cc:485] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:07.381350 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:07.402969 6818 consensus_peers.cc:485] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:07.409911 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:07.434896 9692 consensus_peers.cc:485] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:07.470340 7078 consensus_peers.cc:485] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:07.526732 9690 consensus_peers.cc:485] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:01:07.570565 23276 raft_consensus.cc:482] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:01:07.570724 23276 raft_consensus.cc:504] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:01:07.571223 23276 leader_election.cc:288] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:01:07.572871 6949 leader_election.cc:334] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:07.573035 6949 leader_election.cc:334] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:07.573092 6949 leader_election.cc:302] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:07.573225 23276 raft_consensus.cc:2726] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:07.609213 7078 consensus_peers.cc:485] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:07.625622 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:07.656791 6818 consensus_peers.cc:485] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:07.657706 9690 consensus_peers.cc:485] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:07.669687 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:07.685945 6949 consensus_peers.cc:485] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:07.713493 7211 consensus_peers.cc:485] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:07.789575 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:07.799036 6818 consensus_peers.cc:485] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:07.842767 9692 consensus_peers.cc:485] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:07.844105 7211 consensus_peers.cc:485] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:01:07.877055 23279 raft_consensus.cc:482] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:01:07.877215 23279 raft_consensus.cc:504] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:01:07.877825 23279 leader_election.cc:288] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:01:07.879017 9692 leader_election.cc:334] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:07.879137 9692 leader_election.cc:334] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:07.879190 9692 leader_election.cc:302] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:07.879348 23279 raft_consensus.cc:2726] T d53b8c06a6cb4ba9a6d4b20abeace85d P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:07.916189 7080 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:07.924922 6949 consensus_peers.cc:485] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:07.981982 7078 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.005730 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:08.012925 7211 consensus_peers.cc:485] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:08.065006 7080 consensus_peers.cc:485] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.095258 7080 consensus_peers.cc:485] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.129762 7080 consensus_peers.cc:485] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.155783 9690 consensus_peers.cc:485] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.171386 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:08.216655 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.229046 6949 consensus_peers.cc:485] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.230377 6818 consensus_peers.cc:485] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.262075 9692 consensus_peers.cc:485] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:08.273205 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.287530 7209 consensus_peers.cc:485] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:08.339401 7209 consensus_peers.cc:485] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.342279 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:08.347167 7078 consensus_peers.cc:485] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.357731 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:01:08.371389 23315 raft_consensus.cc:482] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:01:08.371845 23315 raft_consensus.cc:504] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:01:08.372344 23315 leader_election.cc:288] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:01:08.373387 7080 leader_election.cc:334] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:08.373521 7080 leader_election.cc:334] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:01:08.373560 7080 leader_election.cc:302] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:08.373687 23315 raft_consensus.cc:2726] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:08.378949 7078 consensus_peers.cc:485] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.392261 7209 consensus_peers.cc:485] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.392717 9692 consensus_peers.cc:485] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:08.394158 6947 consensus_peers.cc:485] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:01:08.401629 23194 raft_consensus.cc:482] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:01:08.401773 23194 raft_consensus.cc:504] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:01:08.402244 23194 leader_election.cc:288] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:01:08.404318 6816 leader_election.cc:334] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:08.405488 6818 leader_election.cc:334] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:08.405721 6818 leader_election.cc:302] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:08.407107 23306 raft_consensus.cc:2726] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:08.409709 9692 consensus_peers.cc:485] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.419077 7211 consensus_peers.cc:485] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.435142 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.450063 7078 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.484683 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:08.492480 6818 consensus_peers.cc:485] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.522318 6949 consensus_peers.cc:485] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.526263 6947 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:01:08.532583 23306 raft_consensus.cc:482] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:01:08.532727 23306 raft_consensus.cc:504] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:01:08.533282 23306 leader_election.cc:288] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db1a0000aaa346f79c0aa4b6c5105191 (127.0.99.136:42053), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
I20240503 02:01:08.533635 9756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49eabbb8972840759ffa83951cb0b3b7" candidate_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "db1a0000aaa346f79c0aa4b6c5105191" is_pre_election: true
I20240503 02:01:08.533807 9756 raft_consensus.cc:2454] T 49eabbb8972840759ffa83951cb0b3b7 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dcc33cc621a4cbbae155f3230fb7c46 in term 1.
W20240503 02:01:08.534409 6816 leader_election.cc:334] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:08.534621 6818 leader_election.cc:334] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:01:08.534680 6818 leader_election.cc:302] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46, db1a0000aaa346f79c0aa4b6c5105191; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:08.534827 23306 raft_consensus.cc:2726] T 49eabbb8972840759ffa83951cb0b3b7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:08.562662 6816 consensus_peers.cc:485] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.619298 7078 consensus_peers.cc:485] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:01:08.625413 23322 raft_consensus.cc:482] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:01:08.625598 23322 raft_consensus.cc:504] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:01:08.626185 23322 leader_election.cc:288] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:01:08.627375 7078 leader_election.cc:334] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:08.628057 7080 leader_election.cc:334] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:01:08.628127 7080 leader_election.cc:302] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:08.628401 23322 raft_consensus.cc:2726] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:08.649935 7209 consensus_peers.cc:485] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:08.655227 6818 consensus_peers.cc:485] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.680600 6818 consensus_peers.cc:485] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:01:08.700868 23279 raft_consensus.cc:482] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:01:08.701030 23279 raft_consensus.cc:504] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:01:08.701427 23279 leader_election.cc:288] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:01:08.703301 9692 leader_election.cc:334] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:08.703420 9690 leader_election.cc:334] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:01:08.703632 9690 leader_election.cc:302] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:08.703789 23279 raft_consensus.cc:2726] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:08.716444 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:08.733774 6816 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.739245 6949 consensus_peers.cc:485] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.755422 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:08.782162 9692 consensus_peers.cc:485] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.790833 7080 consensus_peers.cc:485] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:08.802729 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.807978 9690 consensus_peers.cc:485] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.836822 7209 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.836895 9692 consensus_peers.cc:485] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.845343 6818 consensus_peers.cc:485] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.845904 6949 consensus_peers.cc:485] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.886993 9690 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.893680 7209 consensus_peers.cc:485] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.901230 6818 consensus_peers.cc:485] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:08.924201 6947 consensus_peers.cc:485] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.947863 7078 consensus_peers.cc:485] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.952086 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:08.984584 6947 consensus_peers.cc:485] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:08.990131 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.032814 7078 consensus_peers.cc:485] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.044107 6949 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.060034 6947 consensus_peers.cc:485] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.060273 6818 consensus_peers.cc:485] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.105412 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.137271 7211 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.146409 6947 consensus_peers.cc:485] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.146682 7211 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.147495 7209 consensus_peers.cc:485] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.178138 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:09.180342 7211 consensus_peers.cc:485] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.182189 6816 consensus_peers.cc:485] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:09.208076 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:01:09.223295 23310 raft_consensus.cc:482] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:01:09.223484 23310 raft_consensus.cc:504] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:01:09.223987 23310 leader_election.cc:288] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:01:09.225617 6947 consensus_peers.cc:485] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.226063 7211 leader_election.cc:334] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:09.226372 7211 leader_election.cc:334] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:09.226426 7211 leader_election.cc:302] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:09.226658 23310 raft_consensus.cc:2726] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:09.229184 6818 consensus_peers.cc:485] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.266043 7211 consensus_peers.cc:485] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.282100 9692 consensus_peers.cc:485] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.290445 7078 consensus_peers.cc:485] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.311489 9692 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.312952 7080 consensus_peers.cc:485] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.339712 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:09.351588 6818 consensus_peers.cc:485] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.376338 6949 consensus_peers.cc:485] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.410838 9692 consensus_peers.cc:485] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.430135 7080 consensus_peers.cc:485] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.459797 9690 consensus_peers.cc:485] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.467715 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:09.514938 7211 consensus_peers.cc:485] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:09.534942 7080 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.548430 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:01:09.557248 23322 raft_consensus.cc:482] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:01:09.557381 23322 raft_consensus.cc:504] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:01:09.557756 23322 leader_election.cc:288] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:01:09.559243 7080 leader_election.cc:334] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:09.559373 7080 leader_election.cc:334] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:09.559428 7080 leader_election.cc:302] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:09.559569 23322 raft_consensus.cc:2726] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:09.577224 6816 consensus_peers.cc:485] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:09.593472 9692 consensus_peers.cc:485] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.594161 9692 consensus_peers.cc:485] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.611446 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:09.612565 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:01:09.624657 6818 consensus_peers.cc:485] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.642175 6816 consensus_peers.cc:485] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.677703 7211 consensus_peers.cc:485] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:01:09.738579 23279 raft_consensus.cc:482] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:01:09.738720 23279 raft_consensus.cc:504] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:01:09.739145 23279 leader_election.cc:288] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:01:09.740288 9692 leader_election.cc:334] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:09.740410 9692 leader_election.cc:334] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:01:09.740445 9692 leader_election.cc:302] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:09.740648 23279 raft_consensus.cc:2726] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:09.743873 7209 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.769462 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:09.790472 7211 consensus_peers.cc:485] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.804585 6818 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.805526 6949 consensus_peers.cc:485] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.914856 6949 consensus_peers.cc:485] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.928509 6818 consensus_peers.cc:485] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:09.937403 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:09.946077 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 02:01:09.966094 23322 raft_consensus.cc:482] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:01:09.966231 23322 raft_consensus.cc:504] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:01:09.966697 23322 leader_election.cc:288] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:01:09.968653 7080 leader_election.cc:334] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:09.968776 7080 leader_election.cc:334] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:01:09.968822 7080 leader_election.cc:302] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:09.969043 23322 raft_consensus.cc:2726] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:10.014031 6818 consensus_peers.cc:485] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:10.075280 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:10.117312 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:10.143489 6818 consensus_peers.cc:485] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:10.154747 6949 consensus_peers.cc:485] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:10.195461 7211 consensus_peers.cc:485] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:10.198783 6818 consensus_peers.cc:485] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:10.200052 9690 consensus_peers.cc:485] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:10.231591 7078 consensus_peers.cc:485] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:10.237109 9690 consensus_peers.cc:485] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:10.245430 7211 consensus_peers.cc:485] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:10.250572 7080 consensus_peers.cc:485] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:10.308049 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:10.310968 7078 consensus_peers.cc:485] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:10.360491 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:10.374210 9692 consensus_peers.cc:485] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:10.377612 7211 consensus_peers.cc:485] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:10.385879 7080 consensus_peers.cc:485] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:10.396322 6818 consensus_peers.cc:485] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:10.457592 7080 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:10.462095 6949 consensus_peers.cc:485] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:10.493613 7078 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:10.574245 9690 consensus_peers.cc:485] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:10.638481 9692 consensus_peers.cc:485] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:10.658442 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:10.663185 7080 consensus_peers.cc:485] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:10.678176 7078 consensus_peers.cc:485] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:10.714445 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:10.742898 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:10.752947 9692 consensus_peers.cc:485] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:10.766855 7209 consensus_peers.cc:485] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:10.777812 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:10.778270 6818 consensus_peers.cc:485] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:01:10.791118 23279 raft_consensus.cc:482] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:01:10.791273 23279 raft_consensus.cc:504] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:01:10.791733 23279 leader_election.cc:288] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:01:10.793450 9690 leader_election.cc:334] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:10.793584 9692 leader_election.cc:334] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:01:10.793665 9692 leader_election.cc:302] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:10.793931 23279 raft_consensus.cc:2726] T 944657ceab07465092a950a8c793ceff P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:10.795212 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:10.810756 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:10.848510 9692 consensus_peers.cc:485] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:10.885529 7209 consensus_peers.cc:485] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:10.913538 6949 consensus_peers.cc:485] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:10.991554 6949 consensus_peers.cc:485] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.014052 7209 consensus_peers.cc:485] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.024680 7080 consensus_peers.cc:485] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:11.025888 6816 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:01:11.030179 23335 raft_consensus.cc:482] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:01:11.030368 23335 raft_consensus.cc:504] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:01:11.030966 23335 leader_election.cc:288] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:01:11.032429 7080 leader_election.cc:334] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:11.032676 7080 leader_election.cc:334] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:11.032742 7080 leader_election.cc:302] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:11.032913 23335 raft_consensus.cc:2726] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:11.033250 7078 leader_election.cc:334] T df13afe0f90c4c24b66026048b5852ec P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:11.033409 6947 consensus_peers.cc:485] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.035575 6818 consensus_peers.cc:485] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.043627 6947 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.044750 7211 consensus_peers.cc:485] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.049376 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.083791 7078 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.137714 7078 consensus_peers.cc:485] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.139145 6816 consensus_peers.cc:485] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.201943 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:11.225562 9692 consensus_peers.cc:485] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.227537 7209 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.250151 9690 consensus_peers.cc:485] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.253863 6949 consensus_peers.cc:485] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.284101 6818 consensus_peers.cc:485] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.300632 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:11.314935 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:11.324018 7209 consensus_peers.cc:485] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:11.327083 7209 consensus_peers.cc:485] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.332559 7078 consensus_peers.cc:485] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.368505 6947 consensus_peers.cc:485] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.398339 9692 consensus_peers.cc:485] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.408390 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.434675 7078 consensus_peers.cc:485] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.435652 9690 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.436093 6818 consensus_peers.cc:485] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:11.440459 7078 consensus_peers.cc:485] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.451978 7078 consensus_peers.cc:485] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.474936 6949 consensus_peers.cc:485] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.493878 6818 consensus_peers.cc:485] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.501672 6816 consensus_peers.cc:485] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:11.520560 6818 consensus_peers.cc:485] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:01:11.538959 23329 raft_consensus.cc:482] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:01:11.539116 23329 raft_consensus.cc:504] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:01:11.539530 23329 leader_election.cc:288] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:01:11.540634 7080 leader_election.cc:334] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:11.540745 7080 leader_election.cc:334] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:01:11.540796 7080 leader_election.cc:302] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:11.540961 23329 raft_consensus.cc:2726] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:11.565521 6947 consensus_peers.cc:485] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.576562 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.582109 6947 consensus_peers.cc:485] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.590735 6949 consensus_peers.cc:485] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.609935 6947 consensus_peers.cc:485] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.614430 6947 consensus_peers.cc:485] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.632222 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.657927 7211 consensus_peers.cc:485] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.676262 7211 consensus_peers.cc:485] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.679728 6818 consensus_peers.cc:485] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.679728 7209 consensus_peers.cc:485] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.687711 6949 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.714601 6818 consensus_peers.cc:485] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.775849 7211 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.791304 7211 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111) [suppressed 200 similar messages]
W20240503 02:01:11.794472 6818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111) [suppressed 253 similar messages]
W20240503 02:01:11.823482 7080 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111) [suppressed 209 similar messages]
W20240503 02:01:11.824658 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:11.838596 7211 consensus_peers.cc:485] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:11.847965 6816 consensus_peers.cc:485] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:11.855340 9692 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.869242 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:11.871234 7211 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.871615 6949 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111) [suppressed 254 similar messages]
W20240503 02:01:11.871743 9692 consensus_peers.cc:485] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.878705 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.881079 6818 consensus_peers.cc:485] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.888120 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:11.888120 9692 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111) [suppressed 180 similar messages]
W20240503 02:01:11.889217 9692 consensus_peers.cc:485] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.929039 7080 consensus_peers.cc:485] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:01:11.936277 23320 raft_consensus.cc:482] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:01:11.936431 23320 raft_consensus.cc:504] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:01:11.936892 23320 leader_election.cc:288] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:01:11.937901 9692 leader_election.cc:334] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:11.938176 9692 leader_election.cc:334] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:01:11.938234 9692 leader_election.cc:302] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:11.938395 23320 raft_consensus.cc:2726] T c4930b3ca84749f6a4cd245411f66075 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:11.950131 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:11.951144 9692 consensus_peers.cc:485] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.955957 7080 consensus_peers.cc:485] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:11.980770 9690 consensus_peers.cc:485] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:12.037708 7080 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:12.108438 9692 consensus_peers.cc:485] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:12.109447 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:01:12.127076 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:12.165153 6949 consensus_peers.cc:485] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:12.170653 7211 consensus_peers.cc:485] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:12.205046 6949 consensus_peers.cc:485] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:01:12.210145 23310 raft_consensus.cc:482] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:01:12.210441 23310 raft_consensus.cc:504] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:01:12.210981 23310 leader_election.cc:288] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:01:12.212064 7209 leader_election.cc:334] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:12.212158 7211 leader_election.cc:334] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:12.212242 7211 leader_election.cc:302] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:12.212380 23310 raft_consensus.cc:2726] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:12.266187 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:12.293843 6816 consensus_peers.cc:485] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:12.329044 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:12.340785 6949 consensus_peers.cc:485] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:12.355469 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:12.379120 6818 consensus_peers.cc:485] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:12.392163 7078 consensus_peers.cc:485] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:12.401364 7211 consensus_peers.cc:485] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:12.410107 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:12.468327 6818 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:12.505024 7209 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:01:12.521737 23341 raft_consensus.cc:482] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:01:12.522061 23341 raft_consensus.cc:504] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:01:12.522879 23341 leader_election.cc:288] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:01:12.524371 6949 leader_election.cc:334] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:12.524503 6949 leader_election.cc:334] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:12.524569 6949 leader_election.cc:302] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:12.524745 23341 raft_consensus.cc:2726] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:12.539268 6818 consensus_peers.cc:485] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:12.557183 9690 consensus_peers.cc:485] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:12.568101 7080 consensus_peers.cc:485] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:12.588733 7211 consensus_peers.cc:485] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:12.608451 6818 consensus_peers.cc:485] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:12.611934 7211 consensus_peers.cc:485] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:12.728654 6818 consensus_peers.cc:485] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:12.729142 6818 consensus_peers.cc:485] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:12.730768 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:12.744007 9690 consensus_peers.cc:485] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:01:12.756019 23329 raft_consensus.cc:482] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:01:12.756237 23329 raft_consensus.cc:504] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:01:12.756879 23329 leader_election.cc:288] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:01:12.758262 7080 leader_election.cc:334] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:12.758545 7078 leader_election.cc:334] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:01:12.758638 7078 leader_election.cc:302] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:12.758818 23329 raft_consensus.cc:2726] T 5a1b7f0005e54894b0bc28c3b988a14a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:12.820865 9692 consensus_peers.cc:485] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:12.837576 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:12.858060 7080 consensus_peers.cc:485] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:12.863965 7080 consensus_peers.cc:485] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:12.872475 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:12.911262 7211 consensus_peers.cc:485] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:12.913108 6949 consensus_peers.cc:485] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:01:12.920730 23349 raft_consensus.cc:482] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:01:12.921027 23349 raft_consensus.cc:504] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:01:12.921638 23349 leader_election.cc:288] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:01:12.923288 7211 leader_election.cc:334] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:12.923441 7211 leader_election.cc:334] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:01:12.923487 7211 leader_election.cc:302] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:12.923679 23310 raft_consensus.cc:2726] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:12.931241 6818 consensus_peers.cc:485] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:12.969532 7078 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:12.978283 6818 consensus_peers.cc:485] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:12.992080 7078 consensus_peers.cc:485] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:13.051645 7080 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.104915 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.144193 7078 consensus_peers.cc:485] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.148319 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.189538 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:13.233889 9692 consensus_peers.cc:485] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:13.244213 9692 consensus_peers.cc:485] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.258543 7209 consensus_peers.cc:485] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.329699 9690 consensus_peers.cc:485] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.340757 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.365478 7209 consensus_peers.cc:485] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:13.400272 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:13.420629 7209 consensus_peers.cc:485] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.427290 6947 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.442143 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:13.461081 9692 consensus_peers.cc:485] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:13.465054 7078 consensus_peers.cc:485] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.487779 6816 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.524353 6949 consensus_peers.cc:485] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.536087 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.572315 7078 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.600569 6816 consensus_peers.cc:485] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.600561 6949 consensus_peers.cc:485] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.652078 7080 consensus_peers.cc:485] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:13.664109 7211 consensus_peers.cc:485] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.664374 6949 consensus_peers.cc:485] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.672777 7209 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.707041 6818 consensus_peers.cc:485] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.778369 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:13.779364 7078 consensus_peers.cc:485] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.784833 6947 consensus_peers.cc:485] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.787321 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.788224 7209 consensus_peers.cc:485] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:13.791957 9690 consensus_peers.cc:485] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.795143 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:13.825059 9692 consensus_peers.cc:485] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.875792 7078 consensus_peers.cc:485] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.882705 7209 consensus_peers.cc:485] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.884208 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:13.915113 6949 consensus_peers.cc:485] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.927707 6947 consensus_peers.cc:485] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.954245 7078 consensus_peers.cc:485] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:13.972326 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:01:13.985416 23354 raft_consensus.cc:482] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:01:13.985612 23354 raft_consensus.cc:504] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:01:13.986231 23354 leader_election.cc:288] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:01:13.987735 6947 leader_election.cc:334] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:13.988052 6949 leader_election.cc:334] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:01:13.988128 6949 leader_election.cc:302] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:13.988363 23354 raft_consensus.cc:2726] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:13.989105 7209 consensus_peers.cc:485] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.000732 6818 consensus_peers.cc:485] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.030009 9692 consensus_peers.cc:485] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.043216 6949 consensus_peers.cc:485] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.070302 6818 consensus_peers.cc:485] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.085791 9690 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.087422 6818 consensus_peers.cc:485] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:14.089372 6818 consensus_peers.cc:485] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.119594 7078 consensus_peers.cc:485] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.122193 6818 consensus_peers.cc:485] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.129215 6947 consensus_peers.cc:485] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.141233 6947 consensus_peers.cc:485] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.196803 6947 consensus_peers.cc:485] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.227167 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:14.247560 6816 consensus_peers.cc:485] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:14.250437 9692 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.268117 7211 consensus_peers.cc:485] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:14.274963 6818 consensus_peers.cc:485] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.275230 6947 consensus_peers.cc:485] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.280032 9692 consensus_peers.cc:485] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.290982 7211 consensus_peers.cc:485] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.291328 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:01:14.348423 23329 raft_consensus.cc:482] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:01:14.348631 23329 raft_consensus.cc:504] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:01:14.349285 23329 leader_election.cc:288] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:01:14.351475 6816 consensus_peers.cc:485] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:14.351666 7080 leader_election.cc:334] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:14.351867 7080 leader_election.cc:334] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:01:14.351923 7080 leader_election.cc:302] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:14.352133 23329 raft_consensus.cc:2726] T 59db016f885347c8bb7e47f7b6f5f8b3 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:14.353082 7080 consensus_peers.cc:485] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.365553 7080 consensus_peers.cc:485] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.367851 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.391268 6949 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.396293 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:14.401178 7211 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.450827 7211 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.468474 6949 consensus_peers.cc:485] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.471838 9692 consensus_peers.cc:485] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.490430 7211 consensus_peers.cc:485] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:01:14.546015 23349 raft_consensus.cc:482] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:01:14.546244 23349 raft_consensus.cc:504] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:01:14.546905 23349 leader_election.cc:288] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:01:14.548381 7211 leader_election.cc:334] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:14.548556 7211 leader_election.cc:334] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:01:14.548625 7211 leader_election.cc:302] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:14.548794 23349 raft_consensus.cc:2726] T c9173280a825443db6088e7141217b29 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:01:14.549715 23360 raft_consensus.cc:482] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:01:14.549820 23360 raft_consensus.cc:504] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } }
I20240503 02:01:14.550303 23360 leader_election.cc:288] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:01:14.551524 9690 leader_election.cc:334] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:14.551578 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.551920 9692 leader_election.cc:334] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:14.552014 9692 leader_election.cc:302] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:14.552150 23360 raft_consensus.cc:2726] T ca35a5b27ddd40108d7c31e711ca37da P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:14.553200 9692 consensus_peers.cc:485] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.555130 7080 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.571015 6818 consensus_peers.cc:485] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.624573 7211 consensus_peers.cc:485] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.652830 9690 consensus_peers.cc:485] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.671104 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:14.687335 9692 consensus_peers.cc:485] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.735044 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:01:14.758576 6949 consensus_peers.cc:485] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.782850 7080 consensus_peers.cc:485] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:01:14.806862 23354 raft_consensus.cc:482] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:01:14.807024 23354 raft_consensus.cc:504] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:01:14.807475 23354 leader_election.cc:288] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:01:14.807951 6816 consensus_peers.cc:485] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.808779 6949 leader_election.cc:334] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:14.809379 6949 leader_election.cc:334] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:14.809546 6949 leader_election.cc:302] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:14.809836 23354 raft_consensus.cc:2726] T aa7f6879b71a412cba659a1c0cbc08d5 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:14.811090 6818 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.811082 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.812737 6949 consensus_peers.cc:485] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.816541 7209 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:14.833565 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:14.835605 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:14.874979 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:14.904619 6818 consensus_peers.cc:485] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:15.000816 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:15.012547 7078 consensus_peers.cc:485] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:15.019426 7211 consensus_peers.cc:485] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:15.022245 6818 consensus_peers.cc:485] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:15.051396 6818 consensus_peers.cc:485] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:15.101697 7211 consensus_peers.cc:485] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:15.130779 7211 consensus_peers.cc:485] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:01:15.181833 23310 raft_consensus.cc:482] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:01:15.182238 23310 raft_consensus.cc:504] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:01:15.183130 23310 leader_election.cc:288] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:01:15.184597 6818 consensus_peers.cc:485] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:15.184780 7209 leader_election.cc:334] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:15.185106 7211 leader_election.cc:334] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:15.185202 7211 leader_election.cc:302] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:15.185446 23310 raft_consensus.cc:2726] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:15.189210 9690 consensus_peers.cc:485] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:15.205111 9690 consensus_peers.cc:485] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:15.249904 6818 consensus_peers.cc:485] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:15.257711 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:15.309181 9692 consensus_peers.cc:485] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:01:15.311029 23334 raft_consensus.cc:482] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:01:15.311210 23334 raft_consensus.cc:504] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:01:15.311733 23334 leader_election.cc:288] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:01:15.314560 6818 leader_election.cc:334] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:15.315294 6816 leader_election.cc:334] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:01:15.315341 6816 leader_election.cc:302] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
W20240503 02:01:15.315332 7211 consensus_peers.cc:485] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:01:15.315558 23334 raft_consensus.cc:2726] T 5089809a9f3846f5b30397b1062d4e55 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:15.328117 6949 consensus_peers.cc:485] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:15.331817 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:15.359021 7078 consensus_peers.cc:485] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:15.369206 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:15.404085 7080 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:15.413864 7080 consensus_peers.cc:485] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:15.457975 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:15.503266 6818 consensus_peers.cc:485] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:15.578891 7080 consensus_peers.cc:485] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:15.581552 6818 consensus_peers.cc:485] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:15.590612 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:15.681011 7209 consensus_peers.cc:485] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:15.695911 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:15.720775 7078 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:15.738021 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:01:15.752650 23361 raft_consensus.cc:482] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:01:15.752895 23361 raft_consensus.cc:504] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:01:15.753520 23361 leader_election.cc:288] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:01:15.754983 6818 leader_election.cc:334] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:15.755221 6818 leader_election.cc:334] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:15.755282 6818 leader_election.cc:302] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:15.755267 23372 raft_consensus.cc:482] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:01:15.755393 23372 raft_consensus.cc:504] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:01:15.755466 23361 raft_consensus.cc:2726] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:01:15.755955 23372 leader_election.cc:288] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:01:15.757033 7209 consensus_peers.cc:485] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:15.757403 7080 leader_election.cc:334] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:15.757714 7078 leader_election.cc:334] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:01:15.757791 7078 leader_election.cc:302] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:15.757910 23372 raft_consensus.cc:2726] T 90600b5313d64a10bf00fc1effd176e6 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:15.782071 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:15.811529 9692 consensus_peers.cc:485] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:15.822402 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:15.828429 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:15.904269 9692 consensus_peers.cc:485] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:15.914731 7080 consensus_peers.cc:485] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:15.937848 7078 consensus_peers.cc:485] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:15.953930 6947 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:15.955859 7209 consensus_peers.cc:485] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:15.958277 7078 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:15.965900 6949 consensus_peers.cc:485] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:01:15.986097 23373 raft_consensus.cc:482] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:01:15.986259 23373 raft_consensus.cc:504] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:01:15.986793 23373 leader_election.cc:288] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:01:15.988236 7080 leader_election.cc:334] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:15.988371 7080 leader_election.cc:334] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:01:15.988412 7080 leader_election.cc:302] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:15.988567 23373 raft_consensus.cc:2726] T 22b6ec13c4e04d6cbade150f0c644e52 P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:15.995301 9690 consensus_peers.cc:485] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.006222 6947 consensus_peers.cc:485] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.012737 9692 consensus_peers.cc:485] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.029626 7078 consensus_peers.cc:485] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.065114 6949 consensus_peers.cc:485] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.114869 9692 consensus_peers.cc:485] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:16.124732 6816 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.139545 7209 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.152516 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.164381 6949 consensus_peers.cc:485] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.170737 6949 consensus_peers.cc:485] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.174963 6947 consensus_peers.cc:485] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.190200 7211 consensus_peers.cc:485] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.192425 7209 consensus_peers.cc:485] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:16.204227 6818 consensus_peers.cc:485] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.235662 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:01:16.249373 23354 raft_consensus.cc:482] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:01:16.249553 23354 raft_consensus.cc:504] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:01:16.250154 23354 leader_election.cc:288] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:01:16.252074 6949 leader_election.cc:334] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:16.252265 6949 leader_election.cc:334] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:16.252321 6949 leader_election.cc:302] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:16.252494 23354 raft_consensus.cc:2726] T 2a6014f0ba4e48efb2944d9aaf1761ab P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:16.253198 9690 consensus_peers.cc:485] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.263994 7078 consensus_peers.cc:485] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.278769 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:16.306247 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:16.327682 7209 consensus_peers.cc:485] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.367312 6818 consensus_peers.cc:485] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.372803 9692 consensus_peers.cc:485] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.376500 6816 consensus_peers.cc:485] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.387480 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.418340 6818 consensus_peers.cc:485] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:16.450505 6949 consensus_peers.cc:485] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.454428 7078 consensus_peers.cc:485] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.502867 6947 consensus_peers.cc:485] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.510193 7078 consensus_peers.cc:485] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.560897 6818 consensus_peers.cc:485] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.567137 6947 consensus_peers.cc:485] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.577628 7209 consensus_peers.cc:485] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.588549 9690 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.605616 7078 consensus_peers.cc:485] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.641224 9692 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.672015 6816 consensus_peers.cc:485] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:16.694450 6947 consensus_peers.cc:485] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.697938 6818 consensus_peers.cc:485] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:01:16.718720 23361 raft_consensus.cc:482] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:01:16.718868 23361 raft_consensus.cc:504] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:01:16.719319 23361 leader_election.cc:288] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:01:16.720659 6816 leader_election.cc:334] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:16.721329 6818 leader_election.cc:334] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:16.721453 6818 leader_election.cc:302] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:16.722220 23361 raft_consensus.cc:2726] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:16.729038 6816 consensus_peers.cc:485] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:16.739146 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:16.767549 7211 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.800588 7080 consensus_peers.cc:485] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.805677 6818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111) [suppressed 252 similar messages]
W20240503 02:01:16.813102 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:16.843079 9692 consensus_peers.cc:485] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.845587 7078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111) [suppressed 209 similar messages]
I20240503 02:01:16.852325 23361 raft_consensus.cc:482] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:01:16.852468 23361 raft_consensus.cc:504] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:01:16.852869 23361 leader_election.cc:288] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:01:16.854607 6818 leader_election.cc:334] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:16.854727 6816 leader_election.cc:334] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:01:16.854823 6816 leader_election.cc:302] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:16.855001 23361 raft_consensus.cc:2726] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:16.858254 7211 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111) [suppressed 210 similar messages]
W20240503 02:01:16.859041 7211 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.869649 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:16.871809 6949 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111) [suppressed 260 similar messages]
W20240503 02:01:16.874018 6947 consensus_peers.cc:485] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.886713 7211 consensus_peers.cc:485] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.887476 6818 consensus_peers.cc:485] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.912859 6818 consensus_peers.cc:485] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.963210 9692 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111) [suppressed 175 similar messages]
W20240503 02:01:16.964031 9692 consensus_peers.cc:485] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.972162 6949 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.984268 7080 consensus_peers.cc:485] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:16.988891 7211 consensus_peers.cc:485] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:17.030648 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:17.044027 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:17.059636 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:17.100167 7211 consensus_peers.cc:485] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:17.119089 6818 consensus_peers.cc:485] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:17.154816 6949 consensus_peers.cc:485] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:17.167049 6818 consensus_peers.cc:485] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:17.178551 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:17.182030 9692 consensus_peers.cc:485] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:17.208505 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:01:17.211659 7080 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:17.226310 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:17.239548 7078 consensus_peers.cc:485] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:17.239331 9692 consensus_peers.cc:485] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:17.241365 7211 consensus_peers.cc:485] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:17.250438 6816 consensus_peers.cc:485] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:17.278793 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:17.303364 9690 consensus_peers.cc:485] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:17.336323 7211 consensus_peers.cc:485] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:17.360169 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:17.393486 6818 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:17.403676 7080 consensus_peers.cc:485] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:17.432927 6949 consensus_peers.cc:485] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:17.453418 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:17.542335 7211 consensus_peers.cc:485] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:17.550604 6949 consensus_peers.cc:485] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:17.560945 6818 consensus_peers.cc:485] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:17.593495 9690 consensus_peers.cc:485] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:17.628317 7211 consensus_peers.cc:485] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:17.655195 9692 consensus_peers.cc:485] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:17.670231 7211 consensus_peers.cc:485] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:17.683653 7209 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:17.703331 6818 consensus_peers.cc:485] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:17.724627 6818 consensus_peers.cc:485] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:17.754766 9690 consensus_peers.cc:485] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:17.764545 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:17.772058 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:17.833786 6818 consensus_peers.cc:485] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:01:17.837457 6730 maintenance_manager.cc:419] P 54aa976bc02b497e8f9115c0d4679ef6: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033385
W20240503 02:01:17.868217 7080 consensus_peers.cc:485] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:01:17.877615 6729 maintenance_manager.cc:643] P 54aa976bc02b497e8f9115c0d4679ef6: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.040s user 0.037s sys 0.000s Metrics: {"bytes_written":107727,"cfile_init":1,"dirs.queue_time_us":259,"dirs.run_cpu_time_us":232,"dirs.run_wall_time_us":964,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1242,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":299,"thread_start_us":206,"threads_started":2,"wal-append.queue_time_us":187}
I20240503 02:01:17.879125 6730 maintenance_manager.cc:419] P 54aa976bc02b497e8f9115c0d4679ef6: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 37422 bytes on disk
I20240503 02:01:17.879949 6729 maintenance_manager.cc:643] P 54aa976bc02b497e8f9115c0d4679ef6: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
W20240503 02:01:17.887454 7078 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:17.891121 6949 consensus_peers.cc:485] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:17.965377 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:17.994673 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:17.996353 7080 consensus_peers.cc:485] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:17.997622 7080 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:01:18.013306 23354 raft_consensus.cc:482] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:01:18.013482 23354 raft_consensus.cc:504] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:01:18.013936 23354 leader_election.cc:288] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:01:18.015261 6949 leader_election.cc:334] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:18.015394 6949 leader_election.cc:334] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:18.015456 6949 leader_election.cc:302] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:18.015612 23354 raft_consensus.cc:2726] T 4a6a5ff679364e2f808bfd5f19b25fda P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:18.040127 6818 consensus_peers.cc:485] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.055790 9692 consensus_peers.cc:485] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:01:18.089215 23361 raft_consensus.cc:482] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:01:18.089625 23361 raft_consensus.cc:504] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:01:18.090226 23361 leader_election.cc:288] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:01:18.091477 6816 leader_election.cc:334] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:18.092248 6818 leader_election.cc:334] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:18.092341 6818 leader_election.cc:302] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:18.092492 23361 raft_consensus.cc:2726] T 442a974500b8458c951993c5b1547820 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:18.101972 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.202021 7209 consensus_peers.cc:485] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:18.202262 6818 consensus_peers.cc:485] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.232252 6947 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.266152 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:18.272384 23382 raft_consensus.cc:482] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:01:18.272541 23382 raft_consensus.cc:504] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:01:18.272965 23382 leader_election.cc:288] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:01:18.274549 9692 leader_election.cc:334] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:18.274681 9692 leader_election.cc:334] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:01:18.274740 9692 leader_election.cc:302] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:18.274909 23382 raft_consensus.cc:2726] T 4eccd41fc53241bb8496354405c92ea6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:01:18.275884 23361 raft_consensus.cc:482] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:01:18.276005 23361 raft_consensus.cc:504] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:01:18.276437 23361 leader_election.cc:288] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:01:18.277714 6818 leader_election.cc:334] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:18.277838 6818 leader_election.cc:334] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:18.277891 6818 leader_election.cc:302] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:18.278127 23361 raft_consensus.cc:2726] T beb175f81d9e48229466503ebf7e3c09 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:18.325588 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.346736 7209 consensus_peers.cc:485] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.348170 9690 consensus_peers.cc:485] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.364240 7209 consensus_peers.cc:485] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.371337 9692 consensus_peers.cc:485] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:18.375689 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:18.379526 6949 consensus_peers.cc:485] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.382985 6947 consensus_peers.cc:485] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.396768 9692 consensus_peers.cc:485] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.399904 7078 consensus_peers.cc:485] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:18.432694 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:18.454493 7209 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.463197 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.465265 7078 consensus_peers.cc:485] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.465927 7078 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.482651 9690 consensus_peers.cc:485] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.483390 6947 consensus_peers.cc:485] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:01:18.501410 23276 raft_consensus.cc:482] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:01:18.501623 23276 raft_consensus.cc:504] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:01:18.502333 23276 leader_election.cc:288] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:01:18.503983 6949 leader_election.cc:334] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:18.504145 6947 leader_election.cc:334] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:01:18.504231 6947 leader_election.cc:302] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:18.504442 23276 raft_consensus.cc:2726] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:18.527225 7080 consensus_peers.cc:485] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:18.529119 6949 consensus_peers.cc:485] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.561007 6949 consensus_peers.cc:485] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.617899 6818 consensus_peers.cc:485] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.624066 7211 consensus_peers.cc:485] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.628077 9692 consensus_peers.cc:485] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.639461 6816 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.640363 9692 consensus_peers.cc:485] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:18.647344 7078 consensus_peers.cc:485] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.662225 6949 consensus_peers.cc:485] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.664297 7209 consensus_peers.cc:485] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.812734 7078 consensus_peers.cc:485] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.834645 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:18.839680 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.847186 7209 consensus_peers.cc:485] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:18.858194 7078 consensus_peers.cc:485] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.860278 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:18.869519 6949 consensus_peers.cc:485] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.877825 6818 consensus_peers.cc:485] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.880239 7078 consensus_peers.cc:485] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.894729 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:18.896293 6816 consensus_peers.cc:485] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:01:18.911283 23373 raft_consensus.cc:482] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:01:18.911509 23373 raft_consensus.cc:504] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:01:18.912078 23373 leader_election.cc:288] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:01:18.913492 7078 leader_election.cc:334] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:18.913861 7080 leader_election.cc:334] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:18.913933 7080 leader_election.cc:302] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:18.914130 23373 raft_consensus.cc:2726] T f20b6a14b8674e6fafefe20ca326d38b P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:18.943477 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:18.952513 7078 consensus_peers.cc:485] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.013348 6947 consensus_peers.cc:485] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.060925 6818 consensus_peers.cc:485] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:19.068673 9692 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.076120 6947 consensus_peers.cc:485] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.083925 9690 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.083909 7209 consensus_peers.cc:485] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.123004 7080 consensus_peers.cc:485] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.124781 6818 consensus_peers.cc:485] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:01:19.133597 23372 raft_consensus.cc:482] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:01:19.133739 23372 raft_consensus.cc:504] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:01:19.134171 23372 leader_election.cc:288] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:01:19.135416 7080 leader_election.cc:334] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:19.135629 7080 leader_election.cc:334] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:19.135696 7080 leader_election.cc:302] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:19.135830 23372 raft_consensus.cc:2726] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:19.152226 6818 consensus_peers.cc:485] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.153317 6818 consensus_peers.cc:485] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.165280 6816 consensus_peers.cc:485] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.214046 6947 consensus_peers.cc:485] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.251611 6818 consensus_peers.cc:485] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.268385 9692 consensus_peers.cc:485] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.280524 7211 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.319286 6949 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.377483 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:19.388068 6816 consensus_peers.cc:485] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:19.391423 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:19.413395 7080 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.421645 7211 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.445117 7211 consensus_peers.cc:485] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.465193 6947 consensus_peers.cc:485] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.467211 6818 consensus_peers.cc:485] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.487728 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:19.493881 9692 consensus_peers.cc:485] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.528116 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.534539 9692 consensus_peers.cc:485] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.565460 7211 consensus_peers.cc:485] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.565584 7211 consensus_peers.cc:485] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.574632 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.574767 9692 consensus_peers.cc:485] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.588124 9690 consensus_peers.cc:485] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.592612 7211 consensus_peers.cc:485] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:19.608178 7080 consensus_peers.cc:485] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.656157 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.661717 6818 consensus_peers.cc:485] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.735857 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:19.769658 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:19.807857 7080 consensus_peers.cc:485] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:19.833175 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:19.851404 7078 consensus_peers.cc:485] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.866753 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:01:19.902477 7211 consensus_peers.cc:485] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.915915 6818 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.939963 7211 consensus_peers.cc:485] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.943357 6949 consensus_peers.cc:485] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.963152 6816 consensus_peers.cc:485] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:19.981679 9690 consensus_peers.cc:485] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:01:20.023049 23402 raft_consensus.cc:482] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:01:20.023381 23402 raft_consensus.cc:504] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } }
I20240503 02:01:20.023939 23402 leader_election.cc:288] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:01:20.025257 6947 leader_election.cc:334] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:20.025516 6949 leader_election.cc:334] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:01:20.025579 6949 leader_election.cc:302] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:20.025727 23402 raft_consensus.cc:2726] T 6673a223a841415783ed8934d13b20ed P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:20.031086 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:20.036597 6949 consensus_peers.cc:485] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:20.073659 7211 consensus_peers.cc:485] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:20.114125 7211 consensus_peers.cc:485] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:20.130061 6818 consensus_peers.cc:485] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:20.134467 6818 consensus_peers.cc:485] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:20.139649 7080 consensus_peers.cc:485] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:20.147104 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:20.194662 6818 consensus_peers.cc:485] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:20.195236 6949 consensus_peers.cc:485] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:20.213227 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:20.247608 9692 consensus_peers.cc:485] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:20.336176 6949 consensus_peers.cc:485] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:20.351619 7209 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:20.384115 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:20.401034 9690 consensus_peers.cc:485] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:20.425311 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:20.435876 7078 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:20.495546 6818 consensus_peers.cc:485] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:20.544502 7080 consensus_peers.cc:485] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:20.553207 7209 consensus_peers.cc:485] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:20.554899 6818 consensus_peers.cc:485] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:20.558439 6949 consensus_peers.cc:485] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:20.614634 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:20.642621 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:20.670357 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:20.689568 7078 consensus_peers.cc:485] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:20.712903 7078 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:20.714596 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:20.746778 6947 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:20.749404 9690 consensus_peers.cc:485] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:20.750190 7080 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:01:20.758137 23400 raft_consensus.cc:482] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:01:20.758286 23400 raft_consensus.cc:504] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:01:20.758716 23400 leader_election.cc:288] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:01:20.759860 9690 leader_election.cc:334] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:20.759972 9692 leader_election.cc:334] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:20.760038 9692 leader_election.cc:302] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:20.760215 23400 raft_consensus.cc:2726] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:20.762902 7078 consensus_peers.cc:485] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:20.765774 9692 consensus_peers.cc:485] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:20.775154 7209 consensus_peers.cc:485] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:20.811945 6818 consensus_peers.cc:485] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:20.866317 6947 consensus_peers.cc:485] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:20.914038 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:20.925375 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:20.939304 7209 consensus_peers.cc:485] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:20.958719 6816 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:20.978643 9692 consensus_peers.cc:485] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:21.008375 7209 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.011188 6947 consensus_peers.cc:485] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.014024 7080 consensus_peers.cc:485] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:21.017189 9692 consensus_peers.cc:485] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:21.018041 6949 consensus_peers.cc:485] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.048774 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:21.060169 6818 consensus_peers.cc:485] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.081066 7209 consensus_peers.cc:485] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:21.084658 6949 consensus_peers.cc:485] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.134555 9692 consensus_peers.cc:485] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.172333 7078 consensus_peers.cc:485] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.204218 9690 consensus_peers.cc:485] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.235275 6949 consensus_peers.cc:485] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.235733 7209 consensus_peers.cc:485] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.238638 9692 consensus_peers.cc:485] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.247391 7078 consensus_peers.cc:485] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.274859 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:21.281549 7209 consensus_peers.cc:485] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.305027 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.310087 7211 consensus_peers.cc:485] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.317831 7078 consensus_peers.cc:485] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.330549 6949 consensus_peers.cc:485] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.336272 7078 consensus_peers.cc:485] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.344801 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:01:21.356143 23397 raft_consensus.cc:482] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:01:21.356323 23397 raft_consensus.cc:504] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:01:21.356839 23397 leader_election.cc:288] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:01:21.358227 7080 leader_election.cc:334] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:21.358330 7078 leader_election.cc:334] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:01:21.358388 7078 leader_election.cc:302] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:21.358513 23397 raft_consensus.cc:2726] T 23ae815010364a5586e60b77ff0a600a P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:21.417704 6816 consensus_peers.cc:485] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:21.424005 7078 consensus_peers.cc:485] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.457830 9692 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.479732 6818 consensus_peers.cc:485] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:21.491425 6818 consensus_peers.cc:485] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.504843 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.518186 6818 consensus_peers.cc:485] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.532110 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:21.563313 6947 consensus_peers.cc:485] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.582930 9690 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.584225 6818 consensus_peers.cc:485] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.595750 6947 consensus_peers.cc:485] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.673146 6818 consensus_peers.cc:485] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.709837 6949 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.765931 7080 consensus_peers.cc:485] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.787524 6816 consensus_peers.cc:485] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:21.805670 7211 consensus_peers.cc:485] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.806250 6816 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111) [suppressed 254 similar messages]
W20240503 02:01:21.819352 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:21.863900 7211 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111) [suppressed 201 similar messages]
W20240503 02:01:21.864682 7211 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.867837 7080 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111) [suppressed 208 similar messages]
W20240503 02:01:21.878729 6947 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111) [suppressed 254 similar messages]
W20240503 02:01:21.880611 6816 consensus_peers.cc:485] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.890731 6818 consensus_peers.cc:485] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.893186 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:21.897379 6947 consensus_peers.cc:485] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.910223 6818 consensus_peers.cc:485] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.930481 9692 consensus_peers.cc:485] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.944161 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:21.947006 6947 consensus_peers.cc:485] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.991309 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:21.999394 9690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111) [suppressed 170 similar messages]
W20240503 02:01:22.000257 9690 consensus_peers.cc:485] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:22.004657 9692 consensus_peers.cc:485] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:22.024935 7211 consensus_peers.cc:485] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:01:22.103998 23276 raft_consensus.cc:482] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:01:22.104176 23276 raft_consensus.cc:504] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:01:22.104658 23276 leader_election.cc:288] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:01:22.107422 6949 leader_election.cc:334] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:22.107717 6949 leader_election.cc:334] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:22.107811 6949 leader_election.cc:302] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:22.107993 23276 raft_consensus.cc:2726] T fd29f3b059874b019153b64ef8aaaa48 P 63331f455be049c1bdd6c23c59ac1bef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 02:01:22.112342 7211 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:22.114941 7080 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:22.142211 7211 consensus_peers.cc:485] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:22.146096 7080 consensus_peers.cc:485] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:22.149587 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:22.150187 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:22.158329 9692 consensus_peers.cc:485] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:22.163564 9692 consensus_peers.cc:485] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:22.213383 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:22.215553 7211 consensus_peers.cc:485] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:22.242674 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:01:22.256037 23334 raft_consensus.cc:482] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7062fbbc2b54fff9e99ec4ea5562c76)
I20240503 02:01:22.256246 23334 raft_consensus.cc:504] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:01:22.256808 23334 leader_election.cc:288] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:01:22.258183 6816 leader_election.cc:334] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:22.258316 6818 leader_election.cc:334] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
I20240503 02:01:22.258376 6818 leader_election.cc:302] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:22.258594 23334 raft_consensus.cc:2726] T a1879ffdfd2e4b26b894542d2f13a7c7 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:22.262578 6818 consensus_peers.cc:485] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:01:22.295208 23416 raft_consensus.cc:482] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:01:22.295444 23416 raft_consensus.cc:504] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "c688f3df56ac4b6d9b0c30fa4cc8ca2a" member_type: VOTER last_known_addr { host: "127.0.99.131" port: 41979 } }
I20240503 02:01:22.296154 23416 leader_election.cc:288] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:01:22.297622 7080 leader_election.cc:334] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:22.297797 7080 leader_election.cc:334] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:22.297854 7080 leader_election.cc:302] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c688f3df56ac4b6d9b0c30fa4cc8ca2a; no voters: 12e8c683db054df3b75c6285b98a5ad8, b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 02:01:22.298051 23416 raft_consensus.cc:2726] T 2755530dc6f04baaaafd5d5214b150af P c688f3df56ac4b6d9b0c30fa4cc8ca2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:22.326140 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:22.326750 7211 consensus_peers.cc:485] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:22.329517 7211 consensus_peers.cc:485] T 1d182ce2c5084082a0990ce8090eb88b P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:01:22.340378 23404 raft_consensus.cc:482] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:01:22.340512 23404 raft_consensus.cc:504] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "1d0e738c34af40fa9950c6f086619b3a" member_type: VOTER last_known_addr { host: "127.0.99.132" port: 36271 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } }
I20240503 02:01:22.340898 23404 leader_election.cc:288] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:01:22.342434 7211 leader_election.cc:334] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:22.342689 7209 leader_election.cc:334] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:01:22.342806 7209 leader_election.cc:302] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d0e738c34af40fa9950c6f086619b3a; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:22.343070 23404 raft_consensus.cc:2726] T b69fd5924f9e4d5dae4dac3bf4b9d161 P 1d0e738c34af40fa9950c6f086619b3a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:22.368765 6818 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:22.380162 7211 consensus_peers.cc:485] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:22.405736 7080 consensus_peers.cc:485] T 8ca4e33409b74f359d3242248652d8bb P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:22.423761 7211 consensus_peers.cc:485] T 9c7396d1b98c4b7ca495ba3ab5dcbbc6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:22.463764 7078 consensus_peers.cc:485] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:22.472432 6818 consensus_peers.cc:485] T 27ac67e7e677405e89c22d22f1e00f93 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:22.475265 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:22.505600 9690 consensus_peers.cc:485] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:22.559144 9692 consensus_peers.cc:485] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:22.588898 6949 consensus_peers.cc:485] T 9d44274085b9441c9f15676d842a0710 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:22.608215 6816 consensus_peers.cc:485] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:22.617133 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:22.635280 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:01:22.650231 6949 consensus_peers.cc:485] T 65b41c24a2504ad386e437c24f1decee P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:22.659551 6818 consensus_peers.cc:485] T b886572c6ec9473bbec744cee8de12f5 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:22.693245 6818 consensus_peers.cc:485] T 54e3860e5cf54663b4538b9e0f96664e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:22.704474 6949 consensus_peers.cc:485] T aa71e80f211f4cce97188f6ac8fd3e8c P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:01:22.725076 23334 raft_consensus.cc:482] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 12e8c683db054df3b75c6285b98a5ad8)
I20240503 02:01:22.725265 23334 raft_consensus.cc:504] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "5dcc33cc621a4cbbae155f3230fb7c46" member_type: VOTER last_known_addr { host: "127.0.99.129" port: 42943 } }
I20240503 02:01:22.726182 23334 leader_election.cc:288] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387)
W20240503 02:01:22.727627 6818 leader_election.cc:334] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:22.727730 6816 leader_election.cc:334] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:01:22.727824 6816 leader_election.cc:302] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dcc33cc621a4cbbae155f3230fb7c46; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:22.727981 23334 raft_consensus.cc:2726] T 9b8976486acf49d5a9d0d496379ee2a6 P 5dcc33cc621a4cbbae155f3230fb7c46 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:22.741792 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:22.791246 6949 consensus_peers.cc:485] T 6fd0378b2cff42a681361930f487d834 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:22.824095 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:22.839890 7209 consensus_peers.cc:485] T 5ec2909a6e484f81aad7b39a374e1ef8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:22.865373 6818 consensus_peers.cc:485] T dec7e3036d384f5db51f50ba93acbbc6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:22.916725 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:22.929759 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:22.939311 7080 consensus_peers.cc:485] T 8295ea038a88485095544cc608b000c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:22.958751 6949 consensus_peers.cc:485] T 1d3be25c6ba341b4a2c5c0007ba2129b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:22.982661 7078 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.008082 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:23.013336 9690 consensus_peers.cc:485] T b3ce9de18bda410aa204bf7c1c32e1b7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:23.066540 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.071193 7078 consensus_peers.cc:485] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.108564 7080 consensus_peers.cc:485] T dfdcdd54606e4320a5d0f506eb6dabf8 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.140601 7209 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:23.146835 6949 consensus_peers.cc:485] T 5cb5072e91534671ad3b4dd85f93fa5b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.154462 7078 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.225036 7209 consensus_peers.cc:485] T b5ccbd83016a460f8870ae8ad44b8bdf P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.231240 6947 consensus_peers.cc:485] T d91783037d994d8882ae6e835b70ae39 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.238139 6818 consensus_peers.cc:485] T 3e824fcc9aac449d9373655b304010e2 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.256686 7080 consensus_peers.cc:485] T c43f13a33fe94033bce9debbae6990c2 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.272358 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:23.289502 7209 consensus_peers.cc:485] T 7046186cd7f540e7b0292da5f422a4a1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.298149 6947 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.321795 6818 consensus_peers.cc:485] T 5bff4d54b9d54807b827bc776dce36c6 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.322224 9690 consensus_peers.cc:485] T 8c97333c56ef4c72ac656d4a0a487c80 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.345234 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:23.363703 6818 consensus_peers.cc:485] T 2065dcd6cde84f9a928e4b817737dd42 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.366657 6947 consensus_peers.cc:485] T fb287629baab4cada193ae57695ef1b5 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.389791 7209 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.413123 7078 consensus_peers.cc:485] T 06bffbde7b9f4b31835f61795e6ca32a P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.420796 7078 consensus_peers.cc:485] T 76e8697ac8624e03a46bddae35f4c9fa P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:23.437723 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.477746 9692 consensus_peers.cc:485] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.521544 7209 consensus_peers.cc:485] T 9cc3435afe5144f9a52a286ac86d234f P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.539426 6949 consensus_peers.cc:485] T aaeebd4c884445ba9f44ac9bfcb9c7ba P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.545617 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:23.552521 9692 consensus_peers.cc:485] T fad1113f16ba45009ae409f57eac4550 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.571802 7209 consensus_peers.cc:485] T 6eeb7c5fa08c48c08c36bf03bd1c85ae P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:23.577416 6949 consensus_peers.cc:485] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.612051 7078 consensus_peers.cc:485] T beae7adf9bb3404ca03709b6ff03c7b5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.612103 6949 consensus_peers.cc:485] T d4fa0337228d4f8cb6b6c3aab437619e P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.679813 6818 consensus_peers.cc:485] T 9b9405669fb640c4a3a8fe2f31af291e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.683832 9692 consensus_peers.cc:485] T 1b6f98b01ead4cea8f362dc283caa901 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:23.703377 7078 consensus_peers.cc:485] T 75175ed25db846b9ba8e8e7b595102ca P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.724043 7078 consensus_peers.cc:485] T d80398c9a264430197262bb42241d58d P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.732113 9692 consensus_peers.cc:485] T b93fe374d2a9496a847480b7824a5937 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:23.732434 7080 consensus_peers.cc:485] T 8db6c1313c0042d48416b4f38fe7166f P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:23.753204 6818 consensus_peers.cc:485] T f454ed0b1a924bec9d08f8740137be9b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:23.755028 7209 consensus_peers.cc:485] T 1eefd7ae55eb4403893f5bd31adc2b4b P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.759342 6816 consensus_peers.cc:485] T c4ac4ad7f9eb4723814f65ce96758c0e P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.765137 7078 consensus_peers.cc:485] T 8ed2793842464443826f83f60fb6dcd1 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.779633 9690 consensus_peers.cc:485] T 719808dd1cb141a8b0538156bddd2761 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.816462 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:23.833352 6949 consensus_peers.cc:485] T 8c3d43c8b2ca42b19616fde1d7eb47a3 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:23.859843 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:01:23.875241 7323 maintenance_manager.cc:419] P 1d0e738c34af40fa9950c6f086619b3a: Scheduling FlushMRSOp(ae9fb6c5833145ba91bbd7c24e68a7e2): perf score=0.033365
I20240503 02:01:23.894649 7061 maintenance_manager.cc:419] P 63331f455be049c1bdd6c23c59ac1bef: Scheduling FlushMRSOp(8c3d43c8b2ca42b19616fde1d7eb47a3): perf score=0.033371
W20240503 02:01:23.909183 6818 consensus_peers.cc:485] T 6364f3919c42498ca47ba114d525279b P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:01:23.918201 23405 raft_consensus.cc:482] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:01:23.918342 23405 raft_consensus.cc:504] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e8c683db054df3b75c6285b98a5ad8" member_type: VOTER last_known_addr { host: "127.0.99.133" port: 37387 } } peers { permanent_uuid: "db1a0000aaa346f79c0aa4b6c5105191" member_type: VOTER last_known_addr { host: "127.0.99.136" port: 42053 } } peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } }
I20240503 02:01:23.918813 23405 leader_election.cc:288] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387), bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471)
W20240503 02:01:23.920893 9690 leader_election.cc:334] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:23.922222 9692 leader_election.cc:334] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
I20240503 02:01:23.922312 9692 leader_election.cc:302] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db1a0000aaa346f79c0aa4b6c5105191; no voters: 12e8c683db054df3b75c6285b98a5ad8, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:23.922451 23405 raft_consensus.cc:2726] T c0e7400f30244b4785c3389c63c7f9d6 P db1a0000aaa346f79c0aa4b6c5105191 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:01:23.930045 7192 maintenance_manager.cc:419] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Scheduling FlushMRSOp(ae9fb6c5833145ba91bbd7c24e68a7e2): perf score=0.033384
I20240503 02:01:23.945071 7213 maintenance_manager.cc:643] P 1d0e738c34af40fa9950c6f086619b3a: FlushMRSOp(ae9fb6c5833145ba91bbd7c24e68a7e2) complete. Timing: real 0.069s user 0.032s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":218,"dirs.queue_time_us":757,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":1061,"drs_written":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4,"lbm_write_time_us":620,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":16768,"thread_start_us":311,"threads_started":3,"wal-append.queue_time_us":32165}
I20240503 02:01:23.947476 7323 maintenance_manager.cc:419] P 1d0e738c34af40fa9950c6f086619b3a: Scheduling FlushMRSOp(822bafcbc6de406997955ff7a1d9b612): perf score=0.033383
I20240503 02:01:23.980603 6951 maintenance_manager.cc:643] P 63331f455be049c1bdd6c23c59ac1bef: FlushMRSOp(8c3d43c8b2ca42b19616fde1d7eb47a3) complete. Timing: real 0.086s user 0.029s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":6852,"dirs.queue_time_us":212,"dirs.run_cpu_time_us":219,"dirs.run_wall_time_us":1289,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1407,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":358,"threads_started":3,"wal-append.queue_time_us":11786}
I20240503 02:01:23.982946 7061 maintenance_manager.cc:419] P 63331f455be049c1bdd6c23c59ac1bef: Scheduling UndoDeltaBlockGCOp(8c3d43c8b2ca42b19616fde1d7eb47a3): 4838 bytes on disk
I20240503 02:01:23.983688 6951 maintenance_manager.cc:643] P 63331f455be049c1bdd6c23c59ac1bef: UndoDeltaBlockGCOp(8c3d43c8b2ca42b19616fde1d7eb47a3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
W20240503 02:01:23.991326 6816 consensus_peers.cc:485] T 8ad236ea5ab347c69026293bef7c04a8 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:01:23.996796 7213 maintenance_manager.cc:643] P 1d0e738c34af40fa9950c6f086619b3a: FlushMRSOp(822bafcbc6de406997955ff7a1d9b612) complete. Timing: real 0.049s user 0.019s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":41,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":1607,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":601,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":133,"threads_started":1,"wal-append.queue_time_us":17075}
I20240503 02:01:23.998975 7323 maintenance_manager.cc:419] P 1d0e738c34af40fa9950c6f086619b3a: Scheduling UndoDeltaBlockGCOp(ae9fb6c5833145ba91bbd7c24e68a7e2): 4923 bytes on disk
W20240503 02:01:23.999531 6947 consensus_peers.cc:485] T 487eafe47df84b468618f473eca37874 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:01:23.999928 7213 maintenance_manager.cc:643] P 1d0e738c34af40fa9950c6f086619b3a: UndoDeltaBlockGCOp(ae9fb6c5833145ba91bbd7c24e68a7e2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240503 02:01:24.001927 7323 maintenance_manager.cc:419] P 1d0e738c34af40fa9950c6f086619b3a: Scheduling UndoDeltaBlockGCOp(822bafcbc6de406997955ff7a1d9b612): 4583 bytes on disk
I20240503 02:01:24.002985 7213 maintenance_manager.cc:643] P 1d0e738c34af40fa9950c6f086619b3a: UndoDeltaBlockGCOp(822bafcbc6de406997955ff7a1d9b612) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":282,"lbm_reads_lt_1ms":4}
I20240503 02:01:24.012161 6930 maintenance_manager.cc:419] P 5dcc33cc621a4cbbae155f3230fb7c46: Scheduling FlushMRSOp(8c3d43c8b2ca42b19616fde1d7eb47a3): perf score=0.033401
I20240503 02:01:24.032886 7082 maintenance_manager.cc:643] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: FlushMRSOp(ae9fb6c5833145ba91bbd7c24e68a7e2) complete. Timing: real 0.102s user 0.031s sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":15545,"dirs.queue_time_us":22138,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":1047,"drs_written":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4,"lbm_write_time_us":626,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":345,"threads_started":3,"wal-append.queue_time_us":22380}
I20240503 02:01:24.038362 7192 maintenance_manager.cc:419] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Scheduling FlushMRSOp(822bafcbc6de406997955ff7a1d9b612): perf score=0.033412
W20240503 02:01:24.047102 9690 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.085314 9692 consensus_peers.cc:485] T eac1ed1eaeb04232b889f32ed1df1b40 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:01:24.097991 6820 maintenance_manager.cc:643] P 5dcc33cc621a4cbbae155f3230fb7c46: FlushMRSOp(8c3d43c8b2ca42b19616fde1d7eb47a3) complete. Timing: real 0.085s user 0.032s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":5210,"dirs.queue_time_us":12855,"dirs.run_cpu_time_us":274,"dirs.run_wall_time_us":1162,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":704,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":304,"threads_started":3,"wal-append.queue_time_us":12018}
I20240503 02:01:24.100387 6930 maintenance_manager.cc:419] P 5dcc33cc621a4cbbae155f3230fb7c46: Scheduling UndoDeltaBlockGCOp(8c3d43c8b2ca42b19616fde1d7eb47a3): 4838 bytes on disk
I20240503 02:01:24.103101 6820 maintenance_manager.cc:643] P 5dcc33cc621a4cbbae155f3230fb7c46: UndoDeltaBlockGCOp(8c3d43c8b2ca42b19616fde1d7eb47a3) complete. Timing: real 0.003s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
W20240503 02:01:24.108685 9692 consensus_peers.cc:485] T 389caa08cfcf48a0b91aa5d083cc7d18 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:01:24.109227 7082 maintenance_manager.cc:643] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: FlushMRSOp(822bafcbc6de406997955ff7a1d9b612) complete. Timing: real 0.071s user 0.015s sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":1517,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":777,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":123,"threads_started":1,"wal-append.queue_time_us":8972}
I20240503 02:01:24.112082 7192 maintenance_manager.cc:419] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Scheduling UndoDeltaBlockGCOp(ae9fb6c5833145ba91bbd7c24e68a7e2): 4923 bytes on disk
W20240503 02:01:24.114635 6818 consensus_peers.cc:485] T f8d3d130f4a44904924f7ccf4316fc75 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:01:24.114619 7082 maintenance_manager.cc:643] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: UndoDeltaBlockGCOp(ae9fb6c5833145ba91bbd7c24e68a7e2) complete. Timing: real 0.002s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4}
I20240503 02:01:24.117873 7192 maintenance_manager.cc:419] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: Scheduling UndoDeltaBlockGCOp(822bafcbc6de406997955ff7a1d9b612): 4583 bytes on disk
I20240503 02:01:24.118539 7082 maintenance_manager.cc:643] P c688f3df56ac4b6d9b0c30fa4cc8ca2a: UndoDeltaBlockGCOp(822bafcbc6de406997955ff7a1d9b612) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
W20240503 02:01:24.125953 6947 consensus_peers.cc:485] T 87652406a8e442e58fe636a85ab72158 P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.127804 7211 consensus_peers.cc:485] T 6d4480c8de204f5397dbdcff91db97f6 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.128054 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:24.132126 7080 consensus_peers.cc:485] T 38f126abb17c4654ae3cabe688d513d5 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.153687 6949 consensus_peers.cc:485] T 85d3e89fac21402387aa812f754cf5bc P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.249876 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:24.251358 6818 consensus_peers.cc:485] T 72e090a6c18e4a13bbb343eb40a07694 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.262271 7211 consensus_peers.cc:485] T d2189e61656a428d8931a82ab94c0198 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.269829 6818 consensus_peers.cc:485] T dcc773c6d5794e26a0003c7d4acc5d21 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.300336 9692 consensus_peers.cc:485] T 31b077072738410f8d7c139ad1b5aa9c P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.302140 6818 consensus_peers.cc:485] T 6b54d409a37e4a64a00b3c366dc22df7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.326289 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
W20240503 02:01:24.326660 6818 consensus_peers.cc:485] T 0c421d2afb394266abce79be05feb8ad P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:01:24.350559 23415 raft_consensus.cc:482] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bb5364d730cb4489a0bfc6c270b7450a)
I20240503 02:01:24.350704 23415 raft_consensus.cc:504] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb5364d730cb4489a0bfc6c270b7450a" member_type: VOTER last_known_addr { host: "127.0.99.134" port: 43471 } } peers { permanent_uuid: "b7062fbbc2b54fff9e99ec4ea5562c76" member_type: VOTER last_known_addr { host: "127.0.99.135" port: 46087 } } peers { permanent_uuid: "63331f455be049c1bdd6c23c59ac1bef" member_type: VOTER last_known_addr { host: "127.0.99.130" port: 37231 } }
I20240503 02:01:24.351137 23415 leader_election.cc:288] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471), b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087)
W20240503 02:01:24.353416 6949 leader_election.cc:334] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:24.354483 6947 leader_election.cc:334] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:01:24.354591 6947 leader_election.cc:302] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63331f455be049c1bdd6c23c59ac1bef; no voters: b7062fbbc2b54fff9e99ec4ea5562c76, bb5364d730cb4489a0bfc6c270b7450a
I20240503 02:01:24.354887 23415 raft_consensus.cc:2726] T 31f5289f1dff4fcaa3492c16e40a3ed7 P 63331f455be049c1bdd6c23c59ac1bef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:01:24.373723 7211 consensus_peers.cc:485] T e811662836164c3fb29a1b12ace7eb41 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.403087 7211 consensus_peers.cc:485] T 9b23d84ca6e343b4b550c3025d2b1f12 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.408417 7080 consensus_peers.cc:485] T 7672f42acd9446ef85eced07729c197b P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.473038 7211 consensus_peers.cc:485] T a5e23eebc629494cbb8262bc123a720a P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.483214 6949 consensus_peers.cc:485] T 43b252f688d74d9b833dcd976b74149b P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.511090 6816 consensus_peers.cc:485] T e1f3ae5943604a868467244691d209f7 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:24.548626 9692 consensus_peers.cc:485] T 5b386bc757e44ca588e3ba6ca76c92b6 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.556807 6947 consensus_peers.cc:485] T 04c3319b8d8e4872a703588d3933b32c P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.558069 9690 consensus_peers.cc:485] T d1df18d990ee43a596a4656dd498b8e7 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.596416 6818 consensus_peers.cc:485] T dccce42f6cd64934808d8c6e5f366e35 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.597173 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.597244 6947 consensus_peers.cc:485] T 96dd9b1f85cf471495dff16d5fa9148a P 63331f455be049c1bdd6c23c59ac1bef -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.598054 6949 consensus_peers.cc:485] T 822cbc676b234eda8770f7b485ebe302 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.607807 9692 consensus_peers.cc:485] T 8feff37a8a6043d2b2f7581deb7c7934 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.612396 6816 consensus_peers.cc:485] T 2084fee295274ad89b407a933fb6070c P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.649942 6818 consensus_peers.cc:485] T 4bd8226227c348d5a5d4d8ff28959d27 P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.678637 7080 consensus_peers.cc:462] T c6833c997d284063955d0b9b09ccb3b7 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111)
W20240503 02:01:24.700328 7211 consensus_peers.cc:485] T 9cc9a609452f4f4aad600381968b9227 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:24.743299 7080 consensus_peers.cc:462] T 822bafcbc6de406997955ff7a1d9b612 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111)
W20240503 02:01:24.743743 9692 consensus_peers.cc:485] T 5097893b51da40059b0ef865d42c4d4a P db1a0000aaa346f79c0aa4b6c5105191 -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.775085 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:01:24.802830 7211 consensus_peers.cc:485] T 02e8cbe02d1b4017909e9bfddaf093d8 P 1d0e738c34af40fa9950c6f086619b3a -> Peer 12e8c683db054df3b75c6285b98a5ad8 (127.0.99.133:37387): Couldn't send request to peer 12e8c683db054df3b75c6285b98a5ad8. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.834815 9692 consensus_peers.cc:485] T 516415bcf2ba44bbb6356731427ee2ec P db1a0000aaa346f79c0aa4b6c5105191 -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.851787 7211 consensus_peers.cc:485] T c029253a832e46a48ae16691fc044502 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.874136 7078 consensus_peers.cc:485] T db656aaa7b104b3182350de320f67839 P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.888870 7211 consensus_peers.cc:485] T d2fbadb911aa4bd7b4db021860c202d3 P 1d0e738c34af40fa9950c6f086619b3a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:01:24.904548 7080 consensus_peers.cc:485] T 848a183ff5f04ca98a852ccc0ab025ff P c688f3df56ac4b6d9b0c30fa4cc8ca2a -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:01:24.922585 6861 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 02:01:24.930038 7254 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 02:01:24.930392 7123 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 02:01:24.940918 7209 consensus_peers.cc:462] T 4600418bf64c4a0d962831c6924c46e1 P 1d0e738c34af40fa9950c6f086619b3a -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:01:24.941857 9735 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 02:01:24.947309 6992 tablet_service.cc:1429] Tablet server has 25 leaders and 0 scanners
W20240503 02:01:24.964800 6816 consensus_peers.cc:485] T f18922f489de49d5b4767afc9ca1b2ed P 5dcc33cc621a4cbbae155f3230fb7c46 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:01:24.991493 9690 consensus_peers.cc:485] T f2d66da6d2e24531aee4aaa6740a7af4 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): Couldn't send request to peer bb5364d730cb4489a0bfc6c270b7450a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:01:24.993602 7061 maintenance_manager.cc:419] P 63331f455be049c1bdd6c23c59ac1bef: Scheduling FlushMRSOp(8ed2793842464443826f83f60fb6dcd1): perf score=0.033349
I20240503 02:01:25.011729 7323 maintenance_manager.cc:419] P 1d0e738c34af40fa9950c6f086619b3a: Scheduling FlushMRSOp(a5e23eebc629494cbb8262bc123a720a): perf score=0.033356
I20240503 02:01:25.012223 6951 maintenance_manager.cc:643] P 63331f455be049c1bdd6c23c59ac1bef: FlushMRSOp(8ed2793842464443826f83f60fb6dcd1) complete. Timing: real 0.018s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":640,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":154,"threads_started":2,"wal-append.queue_time_us":182}
I20240503 02:01:25.016352 7061 maintenance_manager.cc:419] P 63331f455be049c1bdd6c23c59ac1bef: Scheduling UndoDeltaBlockGCOp(8ed2793842464443826f83f60fb6dcd1): 4923 bytes on disk
I20240503 02:01:25.017542 6951 maintenance_manager.cc:643] P 63331f455be049c1bdd6c23c59ac1bef: UndoDeltaBlockGCOp(8ed2793842464443826f83f60fb6dcd1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240503 02:01:25.019922 7061 maintenance_manager.cc:419] P 63331f455be049c1bdd6c23c59ac1bef: Scheduling FlushMRSOp(a5e23eebc629494cbb8262bc123a720a): perf score=0.033355
W20240503 02:01:25.022075 6949 consensus_peers.cc:485] T 622197c9b8664d94bbae183f6f2fd381 P 63331f455be049c1bdd6c23c59ac1bef -> Peer b7062fbbc2b54fff9e99ec4ea5562c76 (127.0.99.135:46087): Couldn't send request to peer b7062fbbc2b54fff9e99ec4ea5562c76. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/rebalancer_tool-test.cc:1203: Failure
Value of: s.ok()
Actual: false
Expected: true
Runtime error: /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 46
Average Replica Count | 22.875000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 1
Maximum | 3
Average | 2.904762
stderr: W20240503 01:59:35.898640 9810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 01:59:35.899019 9810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 01:59:35.903132 9810 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 01:59:36.444276 9810 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 01:59:36.563835 9998 meta_cache.cc:1015] Not found: LookupRpc { table: 'rebalancer_dynamic_table_new_name', partition-key: (<start>), attempt: 1 } failed: table rebalancer_dynamic_table_new_name was deleted: Table deleted at 2024-05-03 01:59:36 UTC
W20240503 01:59:36.942605 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ca54251151134e5998e64c31c165357e: could not find any suitable replica to move from server 1d0e738c34af40fa9950c6f086619b3a to server b7062fbbc2b54fff9e99ec4ea5562c76
W20240503 01:59:36.942840 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3a80d416130e4385870aea6e384ed486: could not find any suitable replica to move from server 63331f455be049c1bdd6c23c59ac1bef to server b7062fbbc2b54fff9e99ec4ea5562c76
W20240503 01:59:36.942930 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ca54251151134e5998e64c31c165357e: could not find any suitable replica to move from server 63331f455be049c1bdd6c23c59ac1bef to server 12e8c683db054df3b75c6285b98a5ad8
W20240503 01:59:36.942973 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 72ea0c008b3e43ccab2d5b224cf8896b: could not find any suitable replica to move from server 12e8c683db054df3b75c6285b98a5ad8 to server 5dcc33cc621a4cbbae155f3230fb7c46
W20240503 01:59:36.943298 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ca54251151134e5998e64c31c165357e: could not find any suitable replica to move from server 1d0e738c34af40fa9950c6f086619b3a to server db1a0000aaa346f79c0aa4b6c5105191
W20240503 01:59:36.943349 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 72ea0c008b3e43ccab2d5b224cf8896b: could not find any suitable replica to move from server bb5364d730cb4489a0bfc6c270b7450a to server c688f3df56ac4b6d9b0c30fa4cc8ca2a
W20240503 01:59:36.943387 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 76d0e687959345af9b8f30fe2c35d640: could not find any suitable replica to move from server c688f3df56ac4b6d9b0c30fa4cc8ca2a to server bb5364d730cb4489a0bfc6c270b7450a
W20240503 01:59:36.943488 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3428bac98e0c49908450b3b51cb9d18c: could not find any suitable replica to move from server 63331f455be049c1bdd6c23c59ac1bef to server b7062fbbc2b54fff9e99ec4ea5562c76
W20240503 01:59:36.943588 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 636b9e88e8fd430595fb3edfb428c61b: could not find any suitable replica to move from server 1d0e738c34af40fa9950c6f086619b3a to server 12e8c683db054df3b75c6285b98a5ad8
W20240503 01:59:36.943636 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f40118346c7d461fb1d445e8dab8b379: could not find any suitable replica to move from server 63331f455be049c1bdd6c23c59ac1bef to server 12e8c683db054df3b75c6285b98a5ad8
W20240503 01:59:36.943678 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d4b0f87e5f7a49e897a3904f2d2e871c: could not find any suitable replica to move from server 63331f455be049c1bdd6c23c59ac1bef to server bb5364d730cb4489a0bfc6c270b7450a
W20240503 01:59:36.943727 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eca57da7c1764bd9a55819692641dac8: could not find any suitable replica to move from server c688f3df56ac4b6d9b0c30fa4cc8ca2a to server b7062fbbc2b54fff9e99ec4ea5562c76
W20240503 01:59:36.943769 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 38e214942900477190e2c3b67ccc5952: could not find any suitable replica to move from server 1d0e738c34af40fa9950c6f086619b3a to server bb5364d730cb4489a0bfc6c270b7450a
W20240503 01:59:36.943828 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8bdc24361c34c13ac0d377e11dc39ae: could not find any suitable replica to move from server 5dcc33cc621a4cbbae155f3230fb7c46 to server db1a0000aaa346f79c0aa4b6c5105191
W20240503 01:59:36.943872 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 06feff591c384a50825820a47b49c8f9: could not find any suitable replica to move from server 5dcc33cc621a4cbbae155f3230fb7c46 to server 12e8c683db054df3b75c6285b98a5ad8
W20240503 01:59:36.943921 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f050ea79f4334dcda01dce5d3e804176: could not find any suitable replica to move from server c688f3df56ac4b6d9b0c30fa4cc8ca2a to server 12e8c683db054df3b75c6285b98a5ad8
W20240503 01:59:36.943964 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3a80d416130e4385870aea6e384ed486: could not find any suitable replica to move from server 63331f455be049c1bdd6c23c59ac1bef to server db1a0000aaa346f79c0aa4b6c5105191
W20240503 01:59:36.944012 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b061a61957e74150a4af26c16c277330: could not find any suitable replica to move from server c688f3df56ac4b6d9b0c30fa4cc8ca2a to server b7062fbbc2b54fff9e99ec4ea5562c76
W20240503 01:59:36.944072 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c0195ea81aa24df495aff19543a2b8eb: could not find any suitable replica to move from server 5dcc33cc621a4cbbae155f3230fb7c46 to server bb5364d730cb4489a0bfc6c270b7450a
W20240503 01:59:36.944108 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ca54251151134e5998e64c31c165357e: could not find any suitable replica to move from server 5dcc33cc621a4cbbae155f3230fb7c46 to server bb5364d730cb4489a0bfc6c270b7450a
W20240503 01:59:36.944168 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d2d157b43e4243979ed58dfe4b2fe341: could not find any suitable replica to move from server 5dcc33cc621a4cbbae155f3230fb7c46 to server db1a0000aaa346f79c0aa4b6c5105191
W20240503 01:59:36.944219 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0f89521df2e14bcd8bbc6f1041196932: could not find any suitable replica to move from server 1d0e738c34af40fa9950c6f086619b3a to server 12e8c683db054df3b75c6285b98a5ad8
W20240503 01:59:36.944260 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table dce2b86d058140cfb0a0d5088d32ec71: could not find any suitable replica to move from server 63331f455be049c1bdd6c23c59ac1bef to server b7062fbbc2b54fff9e99ec4ea5562c76
W20240503 01:59:36.944286 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 45e94c90bda847688db4572a4f9d0c02: could not find any suitable replica to move from server 1d0e738c34af40fa9950c6f086619b3a to server db1a0000aaa346f79c0aa4b6c5105191
W20240503 01:59:36.944343 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9e4998e19d924c9a9fd0a10e74520d26: could not find any suitable replica to move from server c688f3df56ac4b6d9b0c30fa4cc8ca2a to server 12e8c683db054df3b75c6285b98a5ad8
W20240503 01:59:36.944402 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d4b0f87e5f7a49e897a3904f2d2e871c: could not find any suitable replica to move from server 1d0e738c34af40fa9950c6f086619b3a to server bb5364d730cb4489a0bfc6c270b7450a
W20240503 01:59:36.944449 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8bdc24361c34c13ac0d377e11dc39ae: could not find any suitable replica to move from server 63331f455be049c1bdd6c23c59ac1bef to server db1a0000aaa346f79c0aa4b6c5105191
W20240503 01:59:36.944491 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eca57da7c1764bd9a55819692641dac8: could not find any suitable replica to move from server 1d0e738c34af40fa9950c6f086619b3a to server 12e8c683db054df3b75c6285b98a5ad8
W20240503 01:59:36.944525 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ca54251151134e5998e64c31c165357e: could not find any suitable replica to move from server 63331f455be049c1bdd6c23c59ac1bef to server 12e8c683db054df3b75c6285b98a5ad8
W20240503 01:59:36.944576 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 06feff591c384a50825820a47b49c8f9: could not find any suitable replica to move from server 1d0e738c34af40fa9950c6f086619b3a to server b7062fbbc2b54fff9e99ec4ea5562c76
W20240503 01:59:36.944622 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f050ea79f4334dcda01dce5d3e804176: could not find any suitable replica to move from server 5dcc33cc621a4cbbae155f3230fb7c46 to server db1a0000aaa346f79c0aa4b6c5105191
W20240503 01:59:36.944684 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 76d0e687959345af9b8f30fe2c35d640: could not find any suitable replica to move from server 63331f455be049c1bdd6c23c59ac1bef to server bb5364d730cb4489a0bfc6c270b7450a
I20240503 01:59:36.986764 9810 rebalancer_tool.cc:1074] tablet 5ddc47f7144b4cfdbd9a496b5c7562e8: '1d0e738c34af40fa9950c6f086619b3a' -> 'db1a0000aaa346f79c0aa4b6c5105191' move scheduled
I20240503 01:59:37.021620 9810 rebalancer_tool.cc:1074] tablet 9c7396d1b98c4b7ca495ba3ab5dcbbc6: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> '12e8c683db054df3b75c6285b98a5ad8' move scheduled
I20240503 01:59:37.055117 9810 rebalancer_tool.cc:1074] tablet 580e6b4d2b2f4f79b8c2af5e75a3c75f: '63331f455be049c1bdd6c23c59ac1bef' -> 'b7062fbbc2b54fff9e99ec4ea5562c76' move scheduled
I20240503 01:59:37.093386 9810 rebalancer_tool.cc:1074] tablet e811662836164c3fb29a1b12ace7eb41: '5dcc33cc621a4cbbae155f3230fb7c46' -> 'bb5364d730cb4489a0bfc6c270b7450a' move scheduled
I20240503 01:59:37.141862 9810 rebalancer_tool.cc:1074] tablet c6833c997d284063955d0b9b09ccb3b7: '1d0e738c34af40fa9950c6f086619b3a' -> 'db1a0000aaa346f79c0aa4b6c5105191' move scheduled
I20240503 01:59:37.172272 9810 rebalancer_tool.cc:1074] tablet a5e23eebc629494cbb8262bc123a720a: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> '12e8c683db054df3b75c6285b98a5ad8' move scheduled
I20240503 01:59:37.202126 9810 rebalancer_tool.cc:1074] tablet 836882aa1fac4721be3cf096540eedc4: '63331f455be049c1bdd6c23c59ac1bef' -> 'bb5364d730cb4489a0bfc6c270b7450a' move scheduled
I20240503 01:59:37.249385 9810 rebalancer_tool.cc:1074] tablet 5cb5072e91534671ad3b4dd85f93fa5b: '5dcc33cc621a4cbbae155f3230fb7c46' -> 'b7062fbbc2b54fff9e99ec4ea5562c76' move scheduled
I20240503 01:59:37.331763 9810 rebalancer_tool.cc:1074] tablet c35ecf8890b64caaac7e5ea9f6e8f989: '1d0e738c34af40fa9950c6f086619b3a' -> '12e8c683db054df3b75c6285b98a5ad8' move scheduled
I20240503 01:59:37.397492 9810 rebalancer_tool.cc:1074] tablet dec7e3036d384f5db51f50ba93acbbc6: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> 'db1a0000aaa346f79c0aa4b6c5105191' move scheduled
I20240503 01:59:37.437889 9810 rebalancer_tool.cc:1074] tablet ccbd932e20b44dc5aceb86c86d7fda6b: '63331f455be049c1bdd6c23c59ac1bef' -> '12e8c683db054df3b75c6285b98a5ad8' move scheduled
I20240503 01:59:37.482034 9810 rebalancer_tool.cc:1074] tablet 8ed2793842464443826f83f60fb6dcd1: '5dcc33cc621a4cbbae155f3230fb7c46' -> 'bb5364d730cb4489a0bfc6c270b7450a' move scheduled
I20240503 01:59:37.528218 9810 rebalancer_tool.cc:1074] tablet 49b580d648014ce899fee8b3b329c47d: '1d0e738c34af40fa9950c6f086619b3a' -> 'b7062fbbc2b54fff9e99ec4ea5562c76' move scheduled
I20240503 01:59:37.619782 9810 rebalancer_tool.cc:1074] tablet 4600418bf64c4a0d962831c6924c46e1: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> 'db1a0000aaa346f79c0aa4b6c5105191' move scheduled
I20240503 01:59:37.665257 9810 rebalancer_tool.cc:1074] tablet 6d4480c8de204f5397dbdcff91db97f6: '63331f455be049c1bdd6c23c59ac1bef' -> '12e8c683db054df3b75c6285b98a5ad8' move scheduled
I20240503 01:59:37.666647 9810 rebalancer_tool.cc:1084] tablet 6411ac9665194b4f9b9d66f54db0cf25: '5dcc33cc621a4cbbae155f3230fb7c46' -> 'b7062fbbc2b54fff9e99ec4ea5562c76' move ignored: Not found: 6411ac9665194b4f9b9d66f54db0cf25: tablet not found
I20240503 01:59:37.666893 9810 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 01:59:38.019029 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d4b0f87e5f7a49e897a3904f2d2e871c: could not find any suitable replica to move from server 5dcc33cc621a4cbbae155f3230fb7c46 to server b7062fbbc2b54fff9e99ec4ea5562c76
W20240503 01:59:38.019701 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eca57da7c1764bd9a55819692641dac8: could not find any suitable replica to move from server 63331f455be049c1bdd6c23c59ac1bef to server b7062fbbc2b54fff9e99ec4ea5562c76
W20240503 01:59:38.019816 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d4b0f87e5f7a49e897a3904f2d2e871c: could not find any suitable replica to move from server 5dcc33cc621a4cbbae155f3230fb7c46 to server bb5364d730cb4489a0bfc6c270b7450a
W20240503 01:59:38.019898 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cbc39ac1f52f44debbf6d5a5d0edd3ef: could not find any suitable replica to move from server db1a0000aaa346f79c0aa4b6c5105191 to server 1d0e738c34af40fa9950c6f086619b3a
W20240503 01:59:38.019958 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f40118346c7d461fb1d445e8dab8b379: could not find any suitable replica to move from server 63331f455be049c1bdd6c23c59ac1bef to server bb5364d730cb4489a0bfc6c270b7450a
W20240503 01:59:38.020040 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 06feff591c384a50825820a47b49c8f9: could not find any suitable replica to move from server c688f3df56ac4b6d9b0c30fa4cc8ca2a to server db1a0000aaa346f79c0aa4b6c5105191
W20240503 01:59:38.020087 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 72ea0c008b3e43ccab2d5b224cf8896b: could not find any suitable replica to move from server 12e8c683db054df3b75c6285b98a5ad8 to server c688f3df56ac4b6d9b0c30fa4cc8ca2a
W20240503 01:59:38.020123 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 45e94c90bda847688db4572a4f9d0c02: could not find any suitable replica to move from server 63331f455be049c1bdd6c23c59ac1bef to server db1a0000aaa346f79c0aa4b6c5105191
W20240503 01:59:38.020161 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b061a61957e74150a4af26c16c277330: could not find any suitable replica to move from server 1d0e738c34af40fa9950c6f086619b3a to server 12e8c683db054df3b75c6285b98a5ad8
W20240503 01:59:38.020212 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 76d0e687959345af9b8f30fe2c35d640: could not find any suitable replica to move from server 63331f455be049c1bdd6c23c59ac1bef to server b7062fbbc2b54fff9e99ec4ea5562c76
W20240503 01:59:38.020243 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c0195ea81aa24df495aff19543a2b8eb: could not find any suitable replica to move from server 5dcc33cc621a4cbbae155f3230fb7c46 to server db1a0000aaa346f79c0aa4b6c5105191
W20240503 01:59:38.020272 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8bdc24361c34c13ac0d377e11dc39ae: could not find any suitable replica to move from server 1d0e738c34af40fa9950c6f086619b3a to server bb5364d730cb4489a0bfc6c270b7450a
W20240503 01:59:38.020301 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f050ea79f4334dcda01dce5d3e804176: could not find any suitable replica to move from server c688f3df56ac4b6d9b0c30fa4cc8ca2a to server 12e8c683db054df3b75c6285b98a5ad8
W20240503 01:59:38.020347 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d2d157b43e4243979ed58dfe4b2fe341: could not find any suitable replica to move from server 5dcc33cc621a4cbbae155f3230fb7c46 to server b7062fbbc2b54fff9e99ec4ea5562c76
W20240503 01:59:38.020375 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d4b0f87e5f7a49e897a3904f2d2e871c: could not find any suitable replica to move from server 1d0e738c34af40fa9950c6f086619b3a to server 12e8c683db054df3b75c6285b98a5ad8
W20240503 01:59:38.020433 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cbc39ac1f52f44debbf6d5a5d0edd3ef: could not find any suitable replica to move from server 12e8c683db054df3b75c6285b98a5ad8 to server c688f3df56ac4b6d9b0c30fa4cc8ca2a
W20240503 01:59:38.020526 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3a80d416130e4385870aea6e384ed486: could not find any suitable replica to move from server 63331f455be049c1bdd6c23c59ac1bef to server b7062fbbc2b54fff9e99ec4ea5562c76
W20240503 01:59:38.020563 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 38e214942900477190e2c3b67ccc5952: could not find any suitable replica to move from server c688f3df56ac4b6d9b0c30fa4cc8ca2a to server db1a0000aaa346f79c0aa4b6c5105191
W20240503 01:59:38.020612 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 0f89521df2e14bcd8bbc6f1041196932: could not find any suitable replica to move from server 63331f455be049c1bdd6c23c59ac1bef to server 12e8c683db054df3b75c6285b98a5ad8
W20240503 01:59:38.020659 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table dce2b86d058140cfb0a0d5088d32ec71: could not find any suitable replica to move from server 1d0e738c34af40fa9950c6f086619b3a to server bb5364d730cb4489a0bfc6c270b7450a
W20240503 01:59:38.020690 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 9e4998e19d924c9a9fd0a10e74520d26: could not find any suitable replica to move from server c688f3df56ac4b6d9b0c30fa4cc8ca2a to server b7062fbbc2b54fff9e99ec4ea5562c76
W20240503 01:59:38.020720 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3428bac98e0c49908450b3b51cb9d18c: could not find any suitable replica to move from server 5dcc33cc621a4cbbae155f3230fb7c46 to server 12e8c683db054df3b75c6285b98a5ad8
W20240503 01:59:38.020766 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c0195ea81aa24df495aff19543a2b8eb: could not find any suitable replica to move from server 63331f455be049c1bdd6c23c59ac1bef to server 12e8c683db054df3b75c6285b98a5ad8
W20240503 01:59:38.020807 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 45e94c90bda847688db4572a4f9d0c02: could not find any suitable replica to move from server c688f3df56ac4b6d9b0c30fa4cc8ca2a to server db1a0000aaa346f79c0aa4b6c5105191
W20240503 01:59:38.020836 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8bdc24361c34c13ac0d377e11dc39ae: could not find any suitable replica to move from server 5dcc33cc621a4cbbae155f3230fb7c46 to server bb5364d730cb4489a0bfc6c270b7450a
W20240503 01:59:38.020865 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f050ea79f4334dcda01dce5d3e804176: could not find any suitable replica to move from server 1d0e738c34af40fa9950c6f086619b3a to server b7062fbbc2b54fff9e99ec4ea5562c76
W20240503 01:59:38.020891 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d4b0f87e5f7a49e897a3904f2d2e871c: could not find any suitable replica to move from server 63331f455be049c1bdd6c23c59ac1bef to server db1a0000aaa346f79c0aa4b6c5105191
I20240503 01:59:38.074304 9810 rebalancer_tool.cc:1074] tablet 31b077072738410f8d7c139ad1b5aa9c: 'bb5364d730cb4489a0bfc6c270b7450a' -> '5dcc33cc621a4cbbae155f3230fb7c46' move scheduled
I20240503 01:59:38.179705 9810 rebalancer_tool.cc:1074] tablet c029253a832e46a48ae16691fc044502: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> 'b7062fbbc2b54fff9e99ec4ea5562c76' move scheduled
I20240503 01:59:38.213351 9810 rebalancer_tool.cc:1074] tablet 46f4a187af854867bcd509c17ced5cd7: '1d0e738c34af40fa9950c6f086619b3a' -> 'db1a0000aaa346f79c0aa4b6c5105191' move scheduled
I20240503 01:59:38.250870 9810 rebalancer_tool.cc:1074] tablet 8295ea038a88485095544cc608b000c2: '63331f455be049c1bdd6c23c59ac1bef' -> 'b7062fbbc2b54fff9e99ec4ea5562c76' move scheduled
I20240503 01:59:38.284373 9810 rebalancer_tool.cc:1074] tablet db656aaa7b104b3182350de320f67839: '5dcc33cc621a4cbbae155f3230fb7c46' -> 'bb5364d730cb4489a0bfc6c270b7450a' move scheduled
I20240503 01:59:38.742484 9810 rebalancer_tool.cc:1123] tablet 46f4a187af854867bcd509c17ced5cd7: '1d0e738c34af40fa9950c6f086619b3a' -> 'db1a0000aaa346f79c0aa4b6c5105191' move is abandoned: Not found: 46f4a187af854867bcd509c17ced5cd7: tablet not found
I20240503 01:59:39.763986 9810 rebalancer_tool.cc:1123] tablet 5ddc47f7144b4cfdbd9a496b5c7562e8: '1d0e738c34af40fa9950c6f086619b3a' -> 'db1a0000aaa346f79c0aa4b6c5105191' move is abandoned: Not found: 5ddc47f7144b4cfdbd9a496b5c7562e8: tablet not found
I20240503 01:59:40.774367 9810 rebalancer_tool.cc:1123] tablet 836882aa1fac4721be3cf096540eedc4: '63331f455be049c1bdd6c23c59ac1bef' -> 'bb5364d730cb4489a0bfc6c270b7450a' move is abandoned: Not found: 836882aa1fac4721be3cf096540eedc4: tablet not found
E20240503 01:59:41.055354 11466 ksck_remote.cc:630] unable to open table rebalancer_dynamic_table_new_name: Not found: the table does not exist: table_name: "rebalancer_dynamic_table_new_name"
I20240503 01:59:41.064911 9810 rebalancer_tool.cc:1123] tablet c35ecf8890b64caaac7e5ea9f6e8f989: '1d0e738c34af40fa9950c6f086619b3a' -> '12e8c683db054df3b75c6285b98a5ad8' move is abandoned: Not found: c35ecf8890b64caaac7e5ea9f6e8f989: tablet not found
I20240503 01:59:41.552692 9810 rebalancer_tool.cc:1123] tablet 580e6b4d2b2f4f79b8c2af5e75a3c75f: '63331f455be049c1bdd6c23c59ac1bef' -> 'b7062fbbc2b54fff9e99ec4ea5562c76' move is abandoned: Not found: 580e6b4d2b2f4f79b8c2af5e75a3c75f: tablet not found
I20240503 01:59:41.553700 9810 rebalancer_tool.cc:1123] tablet ccbd932e20b44dc5aceb86c86d7fda6b: '63331f455be049c1bdd6c23c59ac1bef' -> '12e8c683db054df3b75c6285b98a5ad8' move is abandoned: Not found: ccbd932e20b44dc5aceb86c86d7fda6b: tablet not found
I20240503 01:59:42.103430 9810 rebalancer_tool.cc:1123] tablet 49b580d648014ce899fee8b3b329c47d: '1d0e738c34af40fa9950c6f086619b3a' -> 'b7062fbbc2b54fff9e99ec4ea5562c76' move is abandoned: Not found: 49b580d648014ce899fee8b3b329c47d: tablet not found
I20240503 01:59:42.376451 9810 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240503 01:59:42.605564 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 607fa4fc037b4c5ba04342cb6d7fd017: could not find any suitable replica to move from server 63331f455be049c1bdd6c23c59ac1bef to server c688f3df56ac4b6d9b0c30fa4cc8ca2a
W20240503 01:59:42.605818 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 76d0e687959345af9b8f30fe2c35d640: could not find any suitable replica to move from server c688f3df56ac4b6d9b0c30fa4cc8ca2a to server b7062fbbc2b54fff9e99ec4ea5562c76
W20240503 01:59:42.605897 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b061a61957e74150a4af26c16c277330: could not find any suitable replica to move from server 1d0e738c34af40fa9950c6f086619b3a to server 12e8c683db054df3b75c6285b98a5ad8
W20240503 01:59:42.605947 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f050ea79f4334dcda01dce5d3e804176: could not find any suitable replica to move from server 5dcc33cc621a4cbbae155f3230fb7c46 to server bb5364d730cb4489a0bfc6c270b7450a
W20240503 01:59:42.605974 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 45e94c90bda847688db4572a4f9d0c02: could not find any suitable replica to move from server 63331f455be049c1bdd6c23c59ac1bef to server db1a0000aaa346f79c0aa4b6c5105191
W20240503 01:59:42.606034 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 72ea0c008b3e43ccab2d5b224cf8896b: could not find any suitable replica to move from server 12e8c683db054df3b75c6285b98a5ad8 to server c688f3df56ac4b6d9b0c30fa4cc8ca2a
W20240503 01:59:42.606082 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 607fa4fc037b4c5ba04342cb6d7fd017: could not find any suitable replica to move from server b7062fbbc2b54fff9e99ec4ea5562c76 to server 5dcc33cc621a4cbbae155f3230fb7c46
W20240503 01:59:42.606112 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d46b94d9de0c43ad876391aee881e04d: could not find any suitable replica to move from server db1a0000aaa346f79c0aa4b6c5105191 to server 1d0e738c34af40fa9950c6f086619b3a
W20240503 01:59:42.606200 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 38e214942900477190e2c3b67ccc5952: could not find any suitable replica to move from server 1d0e738c34af40fa9950c6f086619b3a to server 12e8c683db054df3b75c6285b98a5ad8
W20240503 01:59:42.606261 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b061a61957e74150a4af26c16c277330: could not find any suitable replica to move from server c688f3df56ac4b6d9b0c30fa4cc8ca2a to server b7062fbbc2b54fff9e99ec4ea5562c76
W20240503 01:59:42.606345 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table eca57da7c1764bd9a55819692641dac8: could not find any suitable replica to move from server 1d0e738c34af40fa9950c6f086619b3a to server b7062fbbc2b54fff9e99ec4ea5562c76
W20240503 01:59:42.606426 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table dce2b86d058140cfb0a0d5088d32ec71: could not find any suitable replica to move from server c688f3df56ac4b6d9b0c30fa4cc8ca2a to server db1a0000aaa346f79c0aa4b6c5105191
W20240503 01:59:42.606458 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 06feff591c384a50825820a47b49c8f9: could not find any suitable replica to move from server 1d0e738c34af40fa9950c6f086619b3a to server b7062fbbc2b54fff9e99ec4ea5562c76
W20240503 01:59:42.606504 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3428bac98e0c49908450b3b51cb9d18c: could not find any suitable replica to move from server 5dcc33cc621a4cbbae155f3230fb7c46 to server db1a0000aaa346f79c0aa4b6c5105191
W20240503 01:59:42.606591 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table d2d157b43e4243979ed58dfe4b2fe341: could not find any suitable replica to move from server 63331f455be049c1bdd6c23c59ac1bef to server bb5364d730cb4489a0bfc6c270b7450a
W20240503 01:59:42.606631 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f050ea79f4334dcda01dce5d3e804176: could not find any suitable replica to move from server c688f3df56ac4b6d9b0c30fa4cc8ca2a to server 12e8c683db054df3b75c6285b98a5ad8
W20240503 01:59:42.606662 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 73d68a2fa3594bc4b64c9e73724c9ae0: could not find any suitable replica to move from server db1a0000aaa346f79c0aa4b6c5105191 to server b7062fbbc2b54fff9e99ec4ea5562c76
I20240503 01:59:42.643085 9810 rebalancer_tool.cc:1074] tablet 72e090a6c18e4a13bbb343eb40a07694: '1d0e738c34af40fa9950c6f086619b3a' -> '12e8c683db054df3b75c6285b98a5ad8' move scheduled
I20240503 01:59:42.681928 9810 rebalancer_tool.cc:1074] tablet d2fbadb911aa4bd7b4db021860c202d3: '63331f455be049c1bdd6c23c59ac1bef' -> 'db1a0000aaa346f79c0aa4b6c5105191' move scheduled
I20240503 01:59:42.715248 9810 rebalancer_tool.cc:1074] tablet 2f271b67a6f34eb18c98824ca4b26d17: 'b7062fbbc2b54fff9e99ec4ea5562c76' -> '1d0e738c34af40fa9950c6f086619b3a' move scheduled
I20240503 01:59:42.746475 9810 rebalancer_tool.cc:1074] tablet 76e8697ac8624e03a46bddae35f4c9fa: '63331f455be049c1bdd6c23c59ac1bef' -> 'bb5364d730cb4489a0bfc6c270b7450a' move scheduled
I20240503 01:59:42.794770 9810 rebalancer_tool.cc:1074] tablet 1d182ce2c5084082a0990ce8090eb88b: '12e8c683db054df3b75c6285b98a5ad8' -> '1d0e738c34af40fa9950c6f086619b3a' move scheduled
I20240503 01:59:42.828886 9810 rebalancer_tool.cc:1074] tablet 31f5289f1dff4fcaa3492c16e40a3ed7: 'db1a0000aaa346f79c0aa4b6c5105191' -> '63331f455be049c1bdd6c23c59ac1bef' move scheduled
E20240503 01:59:44.428567 12538 ksck_remote.cc:630] unable to open table rebalancer_dynamic_table_new_name: Not found: the table does not exist: table_name: "rebalancer_dynamic_table_new_name"
I20240503 01:59:44.951321 9810 rebalancer_tool.cc:1136] tablet a5e23eebc629494cbb8262bc123a720a: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> '12e8c683db054df3b75c6285b98a5ad8' move completed: OK
I20240503 01:59:44.957818 9810 rebalancer_tool.cc:1136] tablet dec7e3036d384f5db51f50ba93acbbc6: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> 'db1a0000aaa346f79c0aa4b6c5105191' move completed: OK
I20240503 01:59:44.964932 9810 rebalancer_tool.cc:1136] tablet 6d4480c8de204f5397dbdcff91db97f6: '63331f455be049c1bdd6c23c59ac1bef' -> '12e8c683db054df3b75c6285b98a5ad8' move completed: OK
I20240503 01:59:44.971992 9810 rebalancer_tool.cc:1136] tablet 9c7396d1b98c4b7ca495ba3ab5dcbbc6: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> '12e8c683db054df3b75c6285b98a5ad8' move completed: OK
I20240503 01:59:44.978457 9810 rebalancer_tool.cc:1136] tablet 8295ea038a88485095544cc608b000c2: '63331f455be049c1bdd6c23c59ac1bef' -> 'b7062fbbc2b54fff9e99ec4ea5562c76' move completed: OK
I20240503 01:59:45.252266 9810 rebalancer_tool.cc:1136] tablet db656aaa7b104b3182350de320f67839: '5dcc33cc621a4cbbae155f3230fb7c46' -> 'bb5364d730cb4489a0bfc6c270b7450a' move completed: OK
I20240503 01:59:45.258451 9810 rebalancer_tool.cc:1136] tablet 4600418bf64c4a0d962831c6924c46e1: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> 'db1a0000aaa346f79c0aa4b6c5105191' move completed: OK
I20240503 01:59:45.265965 9810 rebalancer_tool.cc:1136] tablet e811662836164c3fb29a1b12ace7eb41: '5dcc33cc621a4cbbae155f3230fb7c46' -> 'bb5364d730cb4489a0bfc6c270b7450a' move completed: OK
I20240503 01:59:45.272778 9810 rebalancer_tool.cc:1136] tablet 8ed2793842464443826f83f60fb6dcd1: '5dcc33cc621a4cbbae155f3230fb7c46' -> 'bb5364d730cb4489a0bfc6c270b7450a' move completed: OK
I20240503 01:59:45.280185 9810 rebalancer_tool.cc:1136] tablet 31b077072738410f8d7c139ad1b5aa9c: 'bb5364d730cb4489a0bfc6c270b7450a' -> '5dcc33cc621a4cbbae155f3230fb7c46' move completed: OK
I20240503 01:59:45.310813 9810 rebalancer_tool.cc:1074] tablet 87652406a8e442e58fe636a85ab72158: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> 'bb5364d730cb4489a0bfc6c270b7450a' move scheduled
I20240503 01:59:45.355603 9810 rebalancer_tool.cc:1074] tablet 85d3e89fac21402387aa812f754cf5bc: '5dcc33cc621a4cbbae155f3230fb7c46' -> '12e8c683db054df3b75c6285b98a5ad8' move scheduled
I20240503 01:59:45.396612 9810 rebalancer_tool.cc:1074] tablet 8ad236ea5ab347c69026293bef7c04a8: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> 'bb5364d730cb4489a0bfc6c270b7450a' move scheduled
I20240503 01:59:45.441339 9810 rebalancer_tool.cc:1074] tablet dfdcdd54606e4320a5d0f506eb6dabf8: '5dcc33cc621a4cbbae155f3230fb7c46' -> 'b7062fbbc2b54fff9e99ec4ea5562c76' move scheduled
I20240503 01:59:45.472759 9810 rebalancer_tool.cc:1074] tablet 9cc3435afe5144f9a52a286ac86d234f: '12e8c683db054df3b75c6285b98a5ad8' -> '63331f455be049c1bdd6c23c59ac1bef' move scheduled
I20240503 01:59:45.513811 9810 rebalancer_tool.cc:1074] tablet 4be54233e31248cd8deafa6b7845a3f8: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> 'db1a0000aaa346f79c0aa4b6c5105191' move scheduled
I20240503 01:59:45.563438 9810 rebalancer_tool.cc:1074] tablet 1b6f98b01ead4cea8f362dc283caa901: 'bb5364d730cb4489a0bfc6c270b7450a' -> '5dcc33cc621a4cbbae155f3230fb7c46' move scheduled
I20240503 01:59:45.641418 9810 rebalancer_tool.cc:1074] tablet 84812ece4f27484eb008bfbebdbfee84: '12e8c683db054df3b75c6285b98a5ad8' -> '1d0e738c34af40fa9950c6f086619b3a' move scheduled
I20240503 01:59:45.678046 9810 rebalancer_tool.cc:1074] tablet fad1113f16ba45009ae409f57eac4550: 'b7062fbbc2b54fff9e99ec4ea5562c76' -> 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' move scheduled
I20240503 01:59:45.706261 9810 rebalancer_tool.cc:1074] tablet 4e686819aed5407bacc13b3e3a0d4566: 'bb5364d730cb4489a0bfc6c270b7450a' -> '63331f455be049c1bdd6c23c59ac1bef' move scheduled
I20240503 01:59:45.742322 9810 rebalancer_tool.cc:1074] tablet 79efb54e1f8143be868e9ef37f5f925d: '5dcc33cc621a4cbbae155f3230fb7c46' -> 'db1a0000aaa346f79c0aa4b6c5105191' move scheduled
I20240503 01:59:50.313161 9810 rebalancer_tool.cc:1136] tablet d2fbadb911aa4bd7b4db021860c202d3: '63331f455be049c1bdd6c23c59ac1bef' -> 'db1a0000aaa346f79c0aa4b6c5105191' move completed: OK
I20240503 01:59:50.329275 9810 rebalancer_tool.cc:1136] tablet 72e090a6c18e4a13bbb343eb40a07694: '1d0e738c34af40fa9950c6f086619b3a' -> '12e8c683db054df3b75c6285b98a5ad8' move completed: OK
I20240503 01:59:51.100069 9810 rebalancer_tool.cc:1136] tablet c029253a832e46a48ae16691fc044502: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> 'b7062fbbc2b54fff9e99ec4ea5562c76' move completed: OK
I20240503 01:59:51.121369 9810 rebalancer_tool.cc:1136] tablet c6833c997d284063955d0b9b09ccb3b7: '1d0e738c34af40fa9950c6f086619b3a' -> 'db1a0000aaa346f79c0aa4b6c5105191' move completed: OK
I20240503 01:59:51.131321 9810 rebalancer_tool.cc:1136] tablet 5cb5072e91534671ad3b4dd85f93fa5b: '5dcc33cc621a4cbbae155f3230fb7c46' -> 'b7062fbbc2b54fff9e99ec4ea5562c76' move completed: OK
I20240503 01:59:51.171599 9810 rebalancer_tool.cc:1074] tablet ae9fb6c5833145ba91bbd7c24e68a7e2: '5dcc33cc621a4cbbae155f3230fb7c46' -> 'db1a0000aaa346f79c0aa4b6c5105191' move scheduled
I20240503 01:59:51.211439 9810 rebalancer_tool.cc:1074] tablet 54e3860e5cf54663b4538b9e0f96664e: 'b7062fbbc2b54fff9e99ec4ea5562c76' -> 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' move scheduled
I20240503 01:59:52.844717 9810 rebalancer_tool.cc:1136] tablet 1b6f98b01ead4cea8f362dc283caa901: 'bb5364d730cb4489a0bfc6c270b7450a' -> '5dcc33cc621a4cbbae155f3230fb7c46' move completed: OK
I20240503 01:59:52.857592 9810 rebalancer_tool.cc:1136] tablet 79efb54e1f8143be868e9ef37f5f925d: '5dcc33cc621a4cbbae155f3230fb7c46' -> 'db1a0000aaa346f79c0aa4b6c5105191' move completed: OK
I20240503 01:59:52.866411 9810 rebalancer_tool.cc:1136] tablet 4be54233e31248cd8deafa6b7845a3f8: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> 'db1a0000aaa346f79c0aa4b6c5105191' move completed: OK
I20240503 01:59:52.874707 9810 rebalancer_tool.cc:1136] tablet 9cc3435afe5144f9a52a286ac86d234f: '12e8c683db054df3b75c6285b98a5ad8' -> '63331f455be049c1bdd6c23c59ac1bef' move completed: OK
I20240503 01:59:52.883033 9810 rebalancer_tool.cc:1136] tablet dfdcdd54606e4320a5d0f506eb6dabf8: '5dcc33cc621a4cbbae155f3230fb7c46' -> 'b7062fbbc2b54fff9e99ec4ea5562c76' move completed: OK
I20240503 01:59:52.895870 9810 rebalancer_tool.cc:1136] tablet 8ad236ea5ab347c69026293bef7c04a8: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> 'bb5364d730cb4489a0bfc6c270b7450a' move completed: OK
I20240503 01:59:52.903219 9810 rebalancer_tool.cc:1136] tablet 31f5289f1dff4fcaa3492c16e40a3ed7: 'db1a0000aaa346f79c0aa4b6c5105191' -> '63331f455be049c1bdd6c23c59ac1bef' move completed: OK
I20240503 01:59:52.912542 9810 rebalancer_tool.cc:1136] tablet 1d182ce2c5084082a0990ce8090eb88b: '12e8c683db054df3b75c6285b98a5ad8' -> '1d0e738c34af40fa9950c6f086619b3a' move completed: OK
I20240503 01:59:52.920469 9810 rebalancer_tool.cc:1136] tablet 85d3e89fac21402387aa812f754cf5bc: '5dcc33cc621a4cbbae155f3230fb7c46' -> '12e8c683db054df3b75c6285b98a5ad8' move completed: OK
I20240503 01:59:52.929525 9810 rebalancer_tool.cc:1136] tablet 76e8697ac8624e03a46bddae35f4c9fa: '63331f455be049c1bdd6c23c59ac1bef' -> 'bb5364d730cb4489a0bfc6c270b7450a' move completed: OK
I20240503 01:59:52.938607 9810 rebalancer_tool.cc:1136] tablet 87652406a8e442e58fe636a85ab72158: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> 'bb5364d730cb4489a0bfc6c270b7450a' move completed: OK
I20240503 01:59:52.947000 9810 rebalancer_tool.cc:1136] tablet 2f271b67a6f34eb18c98824ca4b26d17: 'b7062fbbc2b54fff9e99ec4ea5562c76' -> '1d0e738c34af40fa9950c6f086619b3a' move completed: OK
I20240503 01:59:53.020722 9810 rebalancer_tool.cc:1074] tablet 822bafcbc6de406997955ff7a1d9b612: '5dcc33cc621a4cbbae155f3230fb7c46' -> 'bb5364d730cb4489a0bfc6c270b7450a' move scheduled
I20240503 01:59:53.073141 9810 rebalancer_tool.cc:1074] tablet dcc773c6d5794e26a0003c7d4acc5d21: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> '12e8c683db054df3b75c6285b98a5ad8' move scheduled
I20240503 01:59:53.151522 9810 rebalancer_tool.cc:1074] tablet 96dd9b1f85cf471495dff16d5fa9148a: '1d0e738c34af40fa9950c6f086619b3a' -> 'bb5364d730cb4489a0bfc6c270b7450a' move scheduled
I20240503 01:59:53.184212 9810 rebalancer_tool.cc:1074] tablet 8c3d43c8b2ca42b19616fde1d7eb47a3: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> 'b7062fbbc2b54fff9e99ec4ea5562c76' move scheduled
I20240503 01:59:53.226508 9810 rebalancer_tool.cc:1074] tablet 23ae815010364a5586e60b77ff0a600a: '5dcc33cc621a4cbbae155f3230fb7c46' -> 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' move scheduled
I20240503 01:59:53.261313 9810 rebalancer_tool.cc:1074] tablet 487eafe47df84b468618f473eca37874: '12e8c683db054df3b75c6285b98a5ad8' -> 'bb5364d730cb4489a0bfc6c270b7450a' move scheduled
I20240503 01:59:53.307925 9810 rebalancer_tool.cc:1074] tablet fb287629baab4cada193ae57695ef1b5: 'b7062fbbc2b54fff9e99ec4ea5562c76' -> 'bb5364d730cb4489a0bfc6c270b7450a' move scheduled
I20240503 01:59:55.997617 9810 rebalancer_tool.cc:1136] tablet 4e686819aed5407bacc13b3e3a0d4566: 'bb5364d730cb4489a0bfc6c270b7450a' -> '63331f455be049c1bdd6c23c59ac1bef' move completed: OK
I20240503 01:59:56.007064 9810 rebalancer_tool.cc:1136] tablet fad1113f16ba45009ae409f57eac4550: 'b7062fbbc2b54fff9e99ec4ea5562c76' -> 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' move completed: OK
I20240503 01:59:56.018457 9810 rebalancer_tool.cc:1136] tablet ae9fb6c5833145ba91bbd7c24e68a7e2: '5dcc33cc621a4cbbae155f3230fb7c46' -> 'db1a0000aaa346f79c0aa4b6c5105191' move completed: OK
I20240503 01:59:56.035465 9810 rebalancer_tool.cc:1136] tablet 84812ece4f27484eb008bfbebdbfee84: '12e8c683db054df3b75c6285b98a5ad8' -> '1d0e738c34af40fa9950c6f086619b3a' move completed: OK
I20240503 01:59:56.079682 9810 rebalancer_tool.cc:1074] tablet 3e824fcc9aac449d9373655b304010e2: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> '5dcc33cc621a4cbbae155f3230fb7c46' move scheduled
I20240503 01:59:57.340219 9810 rebalancer_tool.cc:1136] tablet 23ae815010364a5586e60b77ff0a600a: '5dcc33cc621a4cbbae155f3230fb7c46' -> 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' move completed: OK
I20240503 01:59:57.349861 9810 rebalancer_tool.cc:1136] tablet 8c3d43c8b2ca42b19616fde1d7eb47a3: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> 'b7062fbbc2b54fff9e99ec4ea5562c76' move completed: OK
I20240503 01:59:57.357995 9810 rebalancer_tool.cc:1136] tablet 96dd9b1f85cf471495dff16d5fa9148a: '1d0e738c34af40fa9950c6f086619b3a' -> 'bb5364d730cb4489a0bfc6c270b7450a' move completed: OK
I20240503 01:59:57.368198 9810 rebalancer_tool.cc:1136] tablet dcc773c6d5794e26a0003c7d4acc5d21: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> '12e8c683db054df3b75c6285b98a5ad8' move completed: OK
I20240503 01:59:57.383924 9810 rebalancer_tool.cc:1136] tablet 822bafcbc6de406997955ff7a1d9b612: '5dcc33cc621a4cbbae155f3230fb7c46' -> 'bb5364d730cb4489a0bfc6c270b7450a' move completed: OK
I20240503 01:59:57.396489 9810 rebalancer_tool.cc:1136] tablet 54e3860e5cf54663b4538b9e0f96664e: 'b7062fbbc2b54fff9e99ec4ea5562c76' -> 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' move completed: OK
I20240503 01:59:57.442268 9810 rebalancer_tool.cc:1074] tablet 2065dcd6cde84f9a928e4b817737dd42: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> '12e8c683db054df3b75c6285b98a5ad8' move scheduled
I20240503 01:59:57.543347 9810 rebalancer_tool.cc:1074] tablet 4bd8226227c348d5a5d4d8ff28959d27: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> 'b7062fbbc2b54fff9e99ec4ea5562c76' move scheduled
I20240503 01:59:57.596446 9810 rebalancer_tool.cc:1074] tablet c4ac4ad7f9eb4723814f65ce96758c0e: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> '12e8c683db054df3b75c6285b98a5ad8' move scheduled
I20240503 01:59:58.781498 9810 rebalancer_tool.cc:1136] tablet fb287629baab4cada193ae57695ef1b5: 'b7062fbbc2b54fff9e99ec4ea5562c76' -> 'bb5364d730cb4489a0bfc6c270b7450a' move completed: OK
I20240503 01:59:59.210347 9810 rebalancer_tool.cc:1136] tablet 487eafe47df84b468618f473eca37874: '12e8c683db054df3b75c6285b98a5ad8' -> 'bb5364d730cb4489a0bfc6c270b7450a' move completed: OK
I20240503 01:59:59.942672 9810 rebalancer_tool.cc:1136] tablet 2065dcd6cde84f9a928e4b817737dd42: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> '12e8c683db054df3b75c6285b98a5ad8' move completed: OK
I20240503 01:59:59.955789 9810 rebalancer_tool.cc:1136] tablet 3e824fcc9aac449d9373655b304010e2: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> '5dcc33cc621a4cbbae155f3230fb7c46' move completed: OK
I20240503 01:59:59.969012 9810 rebalancer_tool.cc:1136] tablet c4ac4ad7f9eb4723814f65ce96758c0e: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> '12e8c683db054df3b75c6285b98a5ad8' move completed: OK
I20240503 01:59:59.979518 9810 rebalancer_tool.cc:1136] tablet 4bd8226227c348d5a5d4d8ff28959d27: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> 'b7062fbbc2b54fff9e99ec4ea5562c76' move completed: OK
W20240503 02:00:00.389240 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 04f8fe42403241498bbc9d67e5a5e903: could not find any suitable replica to move from server bb5364d730cb4489a0bfc6c270b7450a to server 63331f455be049c1bdd6c23c59ac1bef
W20240503 02:00:00.389914 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 04f8fe42403241498bbc9d67e5a5e903: could not find any suitable replica to move from server bb5364d730cb4489a0bfc6c270b7450a to server db1a0000aaa346f79c0aa4b6c5105191
W20240503 02:00:00.389995 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6ff0348b1ff345cabc0f4ddf7dd75e21: could not find any suitable replica to move from server db1a0000aaa346f79c0aa4b6c5105191 to server b7062fbbc2b54fff9e99ec4ea5562c76
W20240503 02:00:00.390072 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b11be10400574bd0a4c4768ad34c1fc6: could not find any suitable replica to move from server b7062fbbc2b54fff9e99ec4ea5562c76 to server bb5364d730cb4489a0bfc6c270b7450a
W20240503 02:00:00.390133 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table cb5e07c45e8b46888678a15d79f2cfd9: could not find any suitable replica to move from server bb5364d730cb4489a0bfc6c270b7450a to server 1d0e738c34af40fa9950c6f086619b3a
W20240503 02:00:00.390318 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8ade9b2b5b1a4a56a57cd8db33d169d2: could not find any suitable replica to move from server c688f3df56ac4b6d9b0c30fa4cc8ca2a to server bb5364d730cb4489a0bfc6c270b7450a
W20240503 02:00:00.390360 9810 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 28d2e532bc114d04b0b1e87094387600: could not find any suitable replica to move from server bb5364d730cb4489a0bfc6c270b7450a to server 1d0e738c34af40fa9950c6f086619b3a
I20240503 02:00:00.437736 9810 rebalancer_tool.cc:1074] tablet 8ca4e33409b74f359d3242248652d8bb: '1d0e738c34af40fa9950c6f086619b3a' -> 'db1a0000aaa346f79c0aa4b6c5105191' move scheduled
I20240503 02:00:00.481634 9810 rebalancer_tool.cc:1074] tablet 9b8976486acf49d5a9d0d496379ee2a6: 'b7062fbbc2b54fff9e99ec4ea5562c76' -> '5dcc33cc621a4cbbae155f3230fb7c46' move scheduled
I20240503 02:00:00.525581 9810 rebalancer_tool.cc:1074] tablet 6fd0378b2cff42a681361930f487d834: 'bb5364d730cb4489a0bfc6c270b7450a' -> '63331f455be049c1bdd6c23c59ac1bef' move scheduled
I20240503 02:00:00.571422 9810 rebalancer_tool.cc:1074] tablet 0bbbd17c78bc4e329a3a7c7008a34d86: '12e8c683db054df3b75c6285b98a5ad8' -> 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' move scheduled
I20240503 02:00:00.611788 9810 rebalancer_tool.cc:1074] tablet 4bd8226227c348d5a5d4d8ff28959d27: '1d0e738c34af40fa9950c6f086619b3a' -> 'db1a0000aaa346f79c0aa4b6c5105191' move scheduled
I20240503 02:00:00.660511 9810 rebalancer_tool.cc:1074] tablet 5089809a9f3846f5b30397b1062d4e55: 'b7062fbbc2b54fff9e99ec4ea5562c76' -> '5dcc33cc621a4cbbae155f3230fb7c46' move scheduled
I20240503 02:00:00.696157 9810 rebalancer_tool.cc:1074] tablet b93fe374d2a9496a847480b7824a5937: 'bb5364d730cb4489a0bfc6c270b7450a' -> '63331f455be049c1bdd6c23c59ac1bef' move scheduled
I20240503 02:00:00.746937 9810 rebalancer_tool.cc:1074] tablet 9e21d082e40540d4bb908fc9e90055c9: '12e8c683db054df3b75c6285b98a5ad8' -> 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' move scheduled
I20240503 02:00:00.850710 9810 rebalancer_tool.cc:1074] tablet 76e8697ac8624e03a46bddae35f4c9fa: '1d0e738c34af40fa9950c6f086619b3a' -> 'db1a0000aaa346f79c0aa4b6c5105191' move scheduled
I20240503 02:00:00.909096 9810 rebalancer_tool.cc:1074] tablet d80398c9a264430197262bb42241d58d: 'b7062fbbc2b54fff9e99ec4ea5562c76' -> 'bb5364d730cb4489a0bfc6c270b7450a' move scheduled
I20240503 02:00:00.946810 9810 rebalancer_tool.cc:1074] tablet b3ce9de18bda410aa204bf7c1c32e1b7: '63331f455be049c1bdd6c23c59ac1bef' -> '5dcc33cc621a4cbbae155f3230fb7c46' move scheduled
I20240503 02:00:00.983420 9810 rebalancer_tool.cc:1074] tablet e811662836164c3fb29a1b12ace7eb41: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> '12e8c683db054df3b75c6285b98a5ad8' move scheduled
I20240503 02:00:01.031029 9810 rebalancer_tool.cc:1074] tablet f8d3d130f4a44904924f7ccf4316fc75: 'db1a0000aaa346f79c0aa4b6c5105191' -> '1d0e738c34af40fa9950c6f086619b3a' move scheduled
I20240503 02:00:01.068462 9810 rebalancer_tool.cc:1074] tablet 22b6ec13c4e04d6cbade150f0c644e52: 'bb5364d730cb4489a0bfc6c270b7450a' -> 'b7062fbbc2b54fff9e99ec4ea5562c76' move scheduled
I20240503 02:00:01.193814 9810 rebalancer_tool.cc:1074] tablet e8cb37039be049cf9c4e53bbcfbd80d9: '63331f455be049c1bdd6c23c59ac1bef' -> 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' move scheduled
I20240503 02:00:01.241797 9810 rebalancer_tool.cc:1074] tablet fd29f3b059874b019153b64ef8aaaa48: '5dcc33cc621a4cbbae155f3230fb7c46' -> '12e8c683db054df3b75c6285b98a5ad8' move scheduled
I20240503 02:00:01.298364 9810 rebalancer_tool.cc:1074] tablet aaeebd4c884445ba9f44ac9bfcb9c7ba: 'db1a0000aaa346f79c0aa4b6c5105191' -> '1d0e738c34af40fa9950c6f086619b3a' move scheduled
I20240503 02:00:01.366763 9810 rebalancer_tool.cc:1074] tablet 3e824fcc9aac449d9373655b304010e2: 'bb5364d730cb4489a0bfc6c270b7450a' -> '63331f455be049c1bdd6c23c59ac1bef' move scheduled
I20240503 02:00:01.417286 9810 rebalancer_tool.cc:1074] tablet 7046186cd7f540e7b0292da5f422a4a1: 'b7062fbbc2b54fff9e99ec4ea5562c76' -> 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' move scheduled
I20240503 02:00:01.479511 9810 rebalancer_tool.cc:1074] tablet 5e20b869a89e44f1b0ba854163c31f06: '12e8c683db054df3b75c6285b98a5ad8' -> '5dcc33cc621a4cbbae155f3230fb7c46' move scheduled
I20240503 02:00:11.645534 9810 rebalancer_tool.cc:1136] tablet e8cb37039be049cf9c4e53bbcfbd80d9: '63331f455be049c1bdd6c23c59ac1bef' -> 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' move completed: OK
I20240503 02:00:11.654629 9810 rebalancer_tool.cc:1136] tablet 22b6ec13c4e04d6cbade150f0c644e52: 'bb5364d730cb4489a0bfc6c270b7450a' -> 'b7062fbbc2b54fff9e99ec4ea5562c76' move completed: OK
I20240503 02:00:11.662060 9810 rebalancer_tool.cc:1136] tablet 7046186cd7f540e7b0292da5f422a4a1: 'b7062fbbc2b54fff9e99ec4ea5562c76' -> 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' move completed: OK
I20240503 02:00:11.669198 9810 rebalancer_tool.cc:1136] tablet f8d3d130f4a44904924f7ccf4316fc75: 'db1a0000aaa346f79c0aa4b6c5105191' -> '1d0e738c34af40fa9950c6f086619b3a' move completed: OK
I20240503 02:00:11.676427 9810 rebalancer_tool.cc:1136] tablet b3ce9de18bda410aa204bf7c1c32e1b7: '63331f455be049c1bdd6c23c59ac1bef' -> '5dcc33cc621a4cbbae155f3230fb7c46' move completed: OK
I20240503 02:00:11.684254 9810 rebalancer_tool.cc:1136] tablet d80398c9a264430197262bb42241d58d: 'b7062fbbc2b54fff9e99ec4ea5562c76' -> 'bb5364d730cb4489a0bfc6c270b7450a' move completed: OK
I20240503 02:00:11.692759 9810 rebalancer_tool.cc:1136] tablet 76e8697ac8624e03a46bddae35f4c9fa: '1d0e738c34af40fa9950c6f086619b3a' -> 'db1a0000aaa346f79c0aa4b6c5105191' move completed: OK
I20240503 02:00:11.700058 9810 rebalancer_tool.cc:1136] tablet 9e21d082e40540d4bb908fc9e90055c9: '12e8c683db054df3b75c6285b98a5ad8' -> 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' move completed: OK
I20240503 02:00:11.707639 9810 rebalancer_tool.cc:1136] tablet 5e20b869a89e44f1b0ba854163c31f06: '12e8c683db054df3b75c6285b98a5ad8' -> '5dcc33cc621a4cbbae155f3230fb7c46' move completed: OK
I20240503 02:00:11.714637 9810 rebalancer_tool.cc:1136] tablet b93fe374d2a9496a847480b7824a5937: 'bb5364d730cb4489a0bfc6c270b7450a' -> '63331f455be049c1bdd6c23c59ac1bef' move completed: OK
I20240503 02:00:11.723389 9810 rebalancer_tool.cc:1136] tablet 5089809a9f3846f5b30397b1062d4e55: 'b7062fbbc2b54fff9e99ec4ea5562c76' -> '5dcc33cc621a4cbbae155f3230fb7c46' move completed: OK
I20240503 02:00:11.731446 9810 rebalancer_tool.cc:1136] tablet e811662836164c3fb29a1b12ace7eb41: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> '12e8c683db054df3b75c6285b98a5ad8' move completed: OK
I20240503 02:00:11.739310 9810 rebalancer_tool.cc:1136] tablet 4bd8226227c348d5a5d4d8ff28959d27: '1d0e738c34af40fa9950c6f086619b3a' -> 'db1a0000aaa346f79c0aa4b6c5105191' move completed: OK
I20240503 02:00:11.747390 9810 rebalancer_tool.cc:1136] tablet 0bbbd17c78bc4e329a3a7c7008a34d86: '12e8c683db054df3b75c6285b98a5ad8' -> 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' move completed: OK
I20240503 02:00:11.754876 9810 rebalancer_tool.cc:1136] tablet 6fd0378b2cff42a681361930f487d834: 'bb5364d730cb4489a0bfc6c270b7450a' -> '63331f455be049c1bdd6c23c59ac1bef' move completed: OK
I20240503 02:00:11.767848 9810 rebalancer_tool.cc:1136] tablet 9b8976486acf49d5a9d0d496379ee2a6: 'b7062fbbc2b54fff9e99ec4ea5562c76' -> '5dcc33cc621a4cbbae155f3230fb7c46' move completed: OK
I20240503 02:00:11.776809 9810 rebalancer_tool.cc:1136] tablet aaeebd4c884445ba9f44ac9bfcb9c7ba: 'db1a0000aaa346f79c0aa4b6c5105191' -> '1d0e738c34af40fa9950c6f086619b3a' move completed: OK
I20240503 02:00:11.783998 9810 rebalancer_tool.cc:1136] tablet 8ca4e33409b74f359d3242248652d8bb: '1d0e738c34af40fa9950c6f086619b3a' -> 'db1a0000aaa346f79c0aa4b6c5105191' move completed: OK
I20240503 02:00:11.824496 9810 rebalancer_tool.cc:1074] tablet c4930b3ca84749f6a4cd245411f66075: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> 'b7062fbbc2b54fff9e99ec4ea5562c76' move scheduled
I20240503 02:00:11.865636 9810 rebalancer_tool.cc:1074] tablet 75175ed25db846b9ba8e8e7b595102ca: 'db1a0000aaa346f79c0aa4b6c5105191' -> '1d0e738c34af40fa9950c6f086619b3a' move scheduled
I20240503 02:00:11.898811 9810 rebalancer_tool.cc:1074] tablet 8feff37a8a6043d2b2f7581deb7c7934: '12e8c683db054df3b75c6285b98a5ad8' -> '63331f455be049c1bdd6c23c59ac1bef' move scheduled
I20240503 02:00:11.939591 9810 rebalancer_tool.cc:1074] tablet beb175f81d9e48229466503ebf7e3c09: 'bb5364d730cb4489a0bfc6c270b7450a' -> '5dcc33cc621a4cbbae155f3230fb7c46' move scheduled
I20240503 02:00:11.979022 9810 rebalancer_tool.cc:1074] tablet d2fbadb911aa4bd7b4db021860c202d3: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> 'b7062fbbc2b54fff9e99ec4ea5562c76' move scheduled
I20240503 02:00:12.012557 9810 rebalancer_tool.cc:1074] tablet 7672f42acd9446ef85eced07729c197b: 'db1a0000aaa346f79c0aa4b6c5105191' -> '12e8c683db054df3b75c6285b98a5ad8' move scheduled
I20240503 02:00:12.054648 9810 rebalancer_tool.cc:1074] tablet 2084fee295274ad89b407a933fb6070c: '1d0e738c34af40fa9950c6f086619b3a' -> 'bb5364d730cb4489a0bfc6c270b7450a' move scheduled
I20240503 02:00:12.090481 9810 rebalancer_tool.cc:1074] tablet 2755530dc6f04baaaafd5d5214b150af: '63331f455be049c1bdd6c23c59ac1bef' -> 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' move scheduled
I20240503 02:00:12.154393 9810 rebalancer_tool.cc:1074] tablet 8295ea038a88485095544cc608b000c2: '5dcc33cc621a4cbbae155f3230fb7c46' -> 'db1a0000aaa346f79c0aa4b6c5105191' move scheduled
I20240503 02:00:12.185858 9810 rebalancer_tool.cc:1074] tablet 6364f3919c42498ca47ba114d525279b: 'b7062fbbc2b54fff9e99ec4ea5562c76' -> '1d0e738c34af40fa9950c6f086619b3a' move scheduled
I20240503 02:00:12.226346 9810 rebalancer_tool.cc:1074] tablet aab7d8c559ff446192f471aacaa47584: '12e8c683db054df3b75c6285b98a5ad8' -> '63331f455be049c1bdd6c23c59ac1bef' move scheduled
I20240503 02:00:12.262120 9810 rebalancer_tool.cc:1074] tablet d4fa0337228d4f8cb6b6c3aab437619e: 'bb5364d730cb4489a0bfc6c270b7450a' -> 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' move scheduled
I20240503 02:00:12.303990 9810 rebalancer_tool.cc:1074] tablet 944657ceab07465092a950a8c793ceff: '5dcc33cc621a4cbbae155f3230fb7c46' -> 'db1a0000aaa346f79c0aa4b6c5105191' move scheduled
I20240503 02:00:12.352378 9810 rebalancer_tool.cc:1074] tablet 1b7d65d849e242b4b6c0bc06422001db: 'b7062fbbc2b54fff9e99ec4ea5562c76' -> '1d0e738c34af40fa9950c6f086619b3a' move scheduled
I20240503 02:00:12.417408 9810 rebalancer_tool.cc:1074] tablet 38f126abb17c4654ae3cabe688d513d5: '63331f455be049c1bdd6c23c59ac1bef' -> '12e8c683db054df3b75c6285b98a5ad8' move scheduled
I20240503 02:00:12.464187 9810 rebalancer_tool.cc:1074] tablet b886572c6ec9473bbec744cee8de12f5: 'bb5364d730cb4489a0bfc6c270b7450a' -> '5dcc33cc621a4cbbae155f3230fb7c46' move scheduled
I20240503 02:00:12.511572 9810 rebalancer_tool.cc:1074] tablet f20b6a14b8674e6fafefe20ca326d38b: 'db1a0000aaa346f79c0aa4b6c5105191' -> 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' move scheduled
I20240503 02:00:12.585314 9810 rebalancer_tool.cc:1074] tablet dec7e3036d384f5db51f50ba93acbbc6: '1d0e738c34af40fa9950c6f086619b3a' -> 'b7062fbbc2b54fff9e99ec4ea5562c76' move scheduled
I20240503 02:00:18.860293 9810 rebalancer_tool.cc:1136] tablet fd29f3b059874b019153b64ef8aaaa48: '5dcc33cc621a4cbbae155f3230fb7c46' -> '12e8c683db054df3b75c6285b98a5ad8' move completed: OK
I20240503 02:00:19.842180 9810 rebalancer_tool.cc:1136] tablet d4fa0337228d4f8cb6b6c3aab437619e: 'bb5364d730cb4489a0bfc6c270b7450a' -> 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' move completed: OK
I20240503 02:00:19.849987 9810 rebalancer_tool.cc:1136] tablet aab7d8c559ff446192f471aacaa47584: '12e8c683db054df3b75c6285b98a5ad8' -> '63331f455be049c1bdd6c23c59ac1bef' move completed: OK
I20240503 02:00:19.858873 9810 rebalancer_tool.cc:1136] tablet 6364f3919c42498ca47ba114d525279b: 'b7062fbbc2b54fff9e99ec4ea5562c76' -> '1d0e738c34af40fa9950c6f086619b3a' move completed: OK
I20240503 02:00:19.867082 9810 rebalancer_tool.cc:1136] tablet 2755530dc6f04baaaafd5d5214b150af: '63331f455be049c1bdd6c23c59ac1bef' -> 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' move completed: OK
I20240503 02:00:19.876302 9810 rebalancer_tool.cc:1136] tablet dec7e3036d384f5db51f50ba93acbbc6: '1d0e738c34af40fa9950c6f086619b3a' -> 'b7062fbbc2b54fff9e99ec4ea5562c76' move completed: OK
I20240503 02:00:19.886590 9810 rebalancer_tool.cc:1136] tablet 2084fee295274ad89b407a933fb6070c: '1d0e738c34af40fa9950c6f086619b3a' -> 'bb5364d730cb4489a0bfc6c270b7450a' move completed: OK
I20240503 02:00:19.896821 9810 rebalancer_tool.cc:1136] tablet 7672f42acd9446ef85eced07729c197b: 'db1a0000aaa346f79c0aa4b6c5105191' -> '12e8c683db054df3b75c6285b98a5ad8' move completed: OK
I20240503 02:00:19.908592 9810 rebalancer_tool.cc:1136] tablet d2fbadb911aa4bd7b4db021860c202d3: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> 'b7062fbbc2b54fff9e99ec4ea5562c76' move completed: OK
I20240503 02:00:19.920753 9810 rebalancer_tool.cc:1136] tablet 8295ea038a88485095544cc608b000c2: '5dcc33cc621a4cbbae155f3230fb7c46' -> 'db1a0000aaa346f79c0aa4b6c5105191' move completed: OK
I20240503 02:00:19.930030 9810 rebalancer_tool.cc:1136] tablet beb175f81d9e48229466503ebf7e3c09: 'bb5364d730cb4489a0bfc6c270b7450a' -> '5dcc33cc621a4cbbae155f3230fb7c46' move completed: OK
I20240503 02:00:19.937711 9810 rebalancer_tool.cc:1136] tablet f20b6a14b8674e6fafefe20ca326d38b: 'db1a0000aaa346f79c0aa4b6c5105191' -> 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' move completed: OK
I20240503 02:00:19.946444 9810 rebalancer_tool.cc:1136] tablet 944657ceab07465092a950a8c793ceff: '5dcc33cc621a4cbbae155f3230fb7c46' -> 'db1a0000aaa346f79c0aa4b6c5105191' move completed: OK
I20240503 02:00:19.953752 9810 rebalancer_tool.cc:1136] tablet 8feff37a8a6043d2b2f7581deb7c7934: '12e8c683db054df3b75c6285b98a5ad8' -> '63331f455be049c1bdd6c23c59ac1bef' move completed: OK
I20240503 02:00:19.962060 9810 rebalancer_tool.cc:1136] tablet 75175ed25db846b9ba8e8e7b595102ca: 'db1a0000aaa346f79c0aa4b6c5105191' -> '1d0e738c34af40fa9950c6f086619b3a' move completed: OK
I20240503 02:00:19.970468 9810 rebalancer_tool.cc:1136] tablet c4930b3ca84749f6a4cd245411f66075: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> 'b7062fbbc2b54fff9e99ec4ea5562c76' move completed: OK
I20240503 02:00:19.979696 9810 rebalancer_tool.cc:1136] tablet 3e824fcc9aac449d9373655b304010e2: 'bb5364d730cb4489a0bfc6c270b7450a' -> '63331f455be049c1bdd6c23c59ac1bef' move completed: OK
I20240503 02:00:20.024943 9810 rebalancer_tool.cc:1074] tablet b5329f65c13c47d485cee81cf9cf4f1e: 'db1a0000aaa346f79c0aa4b6c5105191' -> '5dcc33cc621a4cbbae155f3230fb7c46' move scheduled
I20240503 02:00:20.069149 9810 rebalancer_tool.cc:1074] tablet 85d3e89fac21402387aa812f754cf5bc: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> 'b7062fbbc2b54fff9e99ec4ea5562c76' move scheduled
I20240503 02:00:20.136584 9810 rebalancer_tool.cc:1074] tablet 822cbc676b234eda8770f7b485ebe302: '12e8c683db054df3b75c6285b98a5ad8' -> '1d0e738c34af40fa9950c6f086619b3a' move scheduled
I20240503 02:00:20.184187 9810 rebalancer_tool.cc:1074] tablet c4ac4ad7f9eb4723814f65ce96758c0e: '63331f455be049c1bdd6c23c59ac1bef' -> 'bb5364d730cb4489a0bfc6c270b7450a' move scheduled
I20240503 02:00:20.228647 9810 rebalancer_tool.cc:1074] tablet 6eeb7c5fa08c48c08c36bf03bd1c85ae: 'db1a0000aaa346f79c0aa4b6c5105191' -> '5dcc33cc621a4cbbae155f3230fb7c46' move scheduled
I20240503 02:00:20.262917 9810 rebalancer_tool.cc:1074] tablet 72e090a6c18e4a13bbb343eb40a07694: 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' -> 'b7062fbbc2b54fff9e99ec4ea5562c76' move scheduled
I20240503 02:00:20.308753 9810 rebalancer_tool.cc:1074] tablet 9b9405669fb640c4a3a8fe2f31af291e: '12e8c683db054df3b75c6285b98a5ad8' -> '1d0e738c34af40fa9950c6f086619b3a' move scheduled
I20240503 02:00:20.361516 9810 rebalancer_tool.cc:1074] tablet 02e8cbe02d1b4017909e9bfddaf093d8: 'db1a0000aaa346f79c0aa4b6c5105191' -> '63331f455be049c1bdd6c23c59ac1bef' move scheduled
I20240503 02:00:20.399518 9810 rebalancer_tool.cc:1074] tablet 4600418bf64c4a0d962831c6924c46e1: '5dcc33cc621a4cbbae155f3230fb7c46' -> 'bb5364d730cb4489a0bfc6c270b7450a' move scheduled
I20240503 02:00:20.446957 9810 rebalancer_tool.cc:1074] tablet df13afe0f90c4c24b66026048b5852ec: 'b7062fbbc2b54fff9e99ec4ea5562c76' -> 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' move scheduled
I20240503 02:00:20.487238 9810 rebalancer_tool.cc:1074] tablet 822bafcbc6de406997955ff7a1d9b612: '1d0e738c34af40fa9950c6f086619b3a' -> '12e8c683db054df3b75c6285b98a5ad8' move scheduled
I20240503 02:00:20.526758 9810 rebalancer_tool.cc:1074] tablet 49eabbb8972840759ffa83951cb0b3b7: 'db1a0000aaa346f79c0aa4b6c5105191' -> '5dcc33cc621a4cbbae155f3230fb7c46' move scheduled
I20240503 02:00:20.571242 9810 rebalancer_tool.cc:1074] tablet c6833c997d284063955d0b9b09ccb3b7: '63331f455be049c1bdd6c23c59ac1bef' -> 'b7062fbbc2b54fff9e99ec4ea5562c76' move scheduled
I20240503 02:00:20.622056 9810 rebalancer_tool.cc:1074] tablet c43f13a33fe94033bce9debbae6990c2: '12e8c683db054df3b75c6285b98a5ad8' -> 'db1a0000aaa346f79c0aa4b6c5105191' move scheduled
W20240503 02:00:21.589354 22277 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:00:21.490923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.134:43471 (local address 127.0.0.1:48162)
0503 02:00:21.519328 (+ 28405us) negotiation.cc:107] Waiting for socket to connect
0503 02:00:21.519341 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0503 02:00:21.519397 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:00:21.582490 (+ 63093us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":28356}
W20240503 02:00:21.618479 22361 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:00:21.485386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.133:37387 (local address 127.0.0.1:48908)
0503 02:00:21.488478 (+ 3092us) negotiation.cc:107] Waiting for socket to connect
0503 02:00:21.488496 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0503 02:00:21.488593 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:00:21.618285 (+129692us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.133:37387: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":3003,"mutex_wait_us":1371,"thread_start_us":73,"threads_started":1}
W20240503 02:00:21.700383 22359 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:00:21.490513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.135:46087 (local address 127.0.0.1:45752)
0503 02:00:21.518902 (+ 28389us) negotiation.cc:107] Waiting for socket to connect
0503 02:00:21.518915 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0503 02:00:21.518995 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:00:21.700217 (+181222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.135:46087: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":28295}
I20240503 02:00:21.712677 9810 rebalancer_tool.cc:1123] tablet 49eabbb8972840759ffa83951cb0b3b7: 'db1a0000aaa346f79c0aa4b6c5105191' -> '5dcc33cc621a4cbbae155f3230fb7c46' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:21.959560 9810 rebalancer_tool.cc:1123] tablet df13afe0f90c4c24b66026048b5852ec: 'b7062fbbc2b54fff9e99ec4ea5562c76' -> 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.0.99.134:43471: connect: Connection refused (error 111)
I20240503 02:00:24.662940 9810 rebalancer_tool.cc:1136] tablet 38f126abb17c4654ae3cabe688d513d5: '63331f455be049c1bdd6c23c59ac1bef' -> '12e8c683db054df3b75c6285b98a5ad8' move completed: OK
I20240503 02:00:24.673547 9810 rebalancer_tool.cc:1136] tablet 1b7d65d849e242b4b6c0bc06422001db: 'b7062fbbc2b54fff9e99ec4ea5562c76' -> '1d0e738c34af40fa9950c6f086619b3a' move completed: OK
I20240503 02:00:24.683275 9810 rebalancer_tool.cc:1136] tablet b886572c6ec9473bbec744cee8de12f5: 'bb5364d730cb4489a0bfc6c270b7450a' -> '5dcc33cc621a4cbbae155f3230fb7c46' move completed: OK
I20240503 02:01:24.709595 9810 rebalancer_tool.cc:1084] tablet 622197c9b8664d94bbae183f6f2fd381: 'b7062fbbc2b54fff9e99ec4ea5562c76' -> 'c688f3df56ac4b6d9b0c30fa4cc8ca2a' move ignored: Timed out: ChangeConfig RPC to 127.0.99.130:37231 timed out after 60.000s (SENT)
I20240503 02:01:24.709748 9810 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server bb5364d730cb4489a0bfc6c270b7450a (127.0.99.134:43471): unacceptable health status UNAVAILABLE
I20240503 02:01:25.039949 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 6801
I20240503 02:01:25.043130 6951 maintenance_manager.cc:643] P 63331f455be049c1bdd6c23c59ac1bef: FlushMRSOp(a5e23eebc629494cbb8262bc123a720a) complete. Timing: real 0.023s user 0.019s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":236,"dirs.run_wall_time_us":1798,"drs_written":1,"lbm_read_time_us":40,"lbm_reads_lt_1ms":4,"lbm_write_time_us":699,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":116,"threads_started":1,"wal-append.queue_time_us":222}
I20240503 02:01:25.046365 7061 maintenance_manager.cc:419] P 63331f455be049c1bdd6c23c59ac1bef: Scheduling UndoDeltaBlockGCOp(a5e23eebc629494cbb8262bc123a720a): 4838 bytes on disk
I20240503 02:01:25.046975 6951 maintenance_manager.cc:643] P 63331f455be049c1bdd6c23c59ac1bef: UndoDeltaBlockGCOp(a5e23eebc629494cbb8262bc123a720a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240503 02:01:25.051265 7213 maintenance_manager.cc:643] P 1d0e738c34af40fa9950c6f086619b3a: FlushMRSOp(a5e23eebc629494cbb8262bc123a720a) complete. Timing: real 0.039s user 0.013s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":180,"dirs.run_cpu_time_us":293,"dirs.run_wall_time_us":4461,"drs_written":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4,"lbm_write_time_us":576,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":162,"threads_started":2,"wal-append.queue_time_us":6375}
I20240503 02:01:25.054592 7323 maintenance_manager.cc:419] P 1d0e738c34af40fa9950c6f086619b3a: Scheduling UndoDeltaBlockGCOp(a5e23eebc629494cbb8262bc123a720a): 4838 bytes on disk
W20240503 02:01:25.055615 6950 connection.cc:535] client connection to 127.0.99.129:42943 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:01:25.056824 23443 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:01:25.056296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.99.129:42943 (local address 127.0.99.130:43293)
0503 02:01:25.056445 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0503 02:01:25.056461 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0503 02:01:25.056561 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:01:25.056680 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.99.129:42943: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":58}
W20240503 02:01:25.057047 6950 consensus_peers.cc:485] T b5329f65c13c47d485cee81cf9cf4f1e P 63331f455be049c1bdd6c23c59ac1bef -> Peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): Couldn't send request to peer 5dcc33cc621a4cbbae155f3230fb7c46. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:42943: BlockingWrite error: write error: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:01:25.057117 6950 consensus_peers.cc:485] T 5e20b869a89e44f1b0ba854163c31f06 P 63331f455be049c1bdd6c23c59ac1bef -> Peer 5dcc33cc621a4cbbae155f3230fb7c46 (127.0.99.129:42943): Couldn't send request to peer 5dcc33cc621a4cbbae155f3230fb7c46. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.129:42943: BlockingWrite error: write error: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:25.057350 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 6932
I20240503 02:01:25.059242 7213 maintenance_manager.cc:643] P 1d0e738c34af40fa9950c6f086619b3a: UndoDeltaBlockGCOp(a5e23eebc629494cbb8262bc123a720a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":127,"lbm_reads_lt_1ms":4}
I20240503 02:01:25.070199 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 7063
W20240503 02:01:25.081496 9692 connection.cc:535] client connection to 127.0.99.131:41979 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:01:25.081880 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 7194
W20240503 02:01:25.083704 9692 consensus_peers.cc:485] T e8cb37039be049cf9c4e53bbcfbd80d9 P db1a0000aaa346f79c0aa4b6c5105191 -> Peer c688f3df56ac4b6d9b0c30fa4cc8ca2a (127.0.99.131:41979): Couldn't send request to peer c688f3df56ac4b6d9b0c30fa4cc8ca2a. Status: Network error: Client connection negotiation failed: client connection to 127.0.99.131:41979: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:01:25.092703 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 9675
I20240503 02:01:25.102727 398 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu with pid 6710
2024-05-03T02:01:25Z chronyd exiting
I20240503 02:01:25.121821 398 test_util.cc:175] -----------------------------------------------
I20240503 02:01:25.121896 398 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714701525263522-398-0
[ FAILED ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/0, where GetParam() = 4-byte object <00-00 00-00> (127694 ms)
[----------] 1 test from DDLDuringRebalancingTest (127694 ms total)
[----------] 1 test from RebalancingDuringElectionStormTest
[ RUN ] RebalancingDuringElectionStormTest.RoundRobin/0
I20240503 02:01:25.122710 398 test_util.cc:268] Using random seed: 416704147
I20240503 02:01:25.123518 398 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:01:25.123555 398 ts_itest-base.cc:113] --------------
I20240503 02:01:25.123576 398 ts_itest-base.cc:114] 8 tablet servers
I20240503 02:01:25.123594 398 ts_itest-base.cc:115] 3 replicas per TS
I20240503 02:01:25.123621 398 ts_itest-base.cc:116] --------------
2024-05-03T02:01:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:01:25Z Disabled control of system clock
I20240503 02:01:25.132591 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:42407
--webserver_interface=127.0.99.190
--webserver_port=0
--builtin_ntp_servers=127.0.99.148:39877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.99.190:42407
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240503 02:01:25.253299 23513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:01:25.253609 23513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:01:25.253684 23513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:01:25.257035 23513 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:01:25.257143 23513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:01:25.257181 23513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:01:25.257231 23513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:01:25.257272 23513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:01:25.261309 23513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.99.190:42407
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.99.190:42407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.99.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.23513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:01:25.262362 23513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:01:25.263609 23513 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:01:25.271894 23521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:01:25.272123 23519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:01:25.273241 23518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:01:25.274349 23513 server_base.cc:1034] running on GCE node
I20240503 02:01:25.274991 23513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:01:25.276003 23513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:01:25.277204 23513 hybrid_clock.cc:648] HybridClock initialized: now 1714701685277107 us; error 102 us; skew 500 ppm
I20240503 02:01:25.279623 23513 webserver.cc:460] Webserver started at http://127.0.99.190:33347/ using document root <none> and password file <none>
I20240503 02:01:25.280205 23513 fs_manager.cc:359] Metadata directory not provided
I20240503 02:01:25.280267 23513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:01:25.280432 23513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:01:25.282217 23513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "cba31bc073e647d7b48fe47ea5d71c41"
format_stamp: "Formatted at 2024-05-03 02:01:25 on dist-test-slave-w9tw"
I20240503 02:01:25.282868 23513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "cba31bc073e647d7b48fe47ea5d71c41"
format_stamp: "Formatted at 2024-05-03 02:01:25 on dist-test-slave-w9tw"
I20240503 02:01:25.286752 23513 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240503 02:01:25.289224 23527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:01:25.290444 23513 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 02:01:25.290623 23513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal
uuid: "cba31bc073e647d7b48fe47ea5d71c41"
format_stamp: "Formatted at 2024-05-03 02:01:25 on dist-test-slave-w9tw"
I20240503 02:01:25.290732 23513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:01:25.311376 23513 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:01:25.312070 23513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:01:25.312229 23513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:01:25.319358 23513 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.190:42407
I20240503 02:01:25.319368 23579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.190:42407 every 8 connection(s)
I20240503 02:01:25.320446 23513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:01:25.323344 23580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:01:25.329373 23580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cba31bc073e647d7b48fe47ea5d71c41: Bootstrap starting.
I20240503 02:01:25.329463 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 23513
I20240503 02:01:25.329573 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:01:25.332021 23580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cba31bc073e647d7b48fe47ea5d71c41: Neither blocks nor log segments found. Creating new log.
I20240503 02:01:25.332743 23580 log.cc:824] T 00000000000000000000000000000000 P cba31bc073e647d7b48fe47ea5d71c41: Log is configured to *not* fsync() on all Append() calls
I20240503 02:01:25.334510 23580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cba31bc073e647d7b48fe47ea5d71c41: No bootstrap required, opened a new log
I20240503 02:01:25.337497 23580 raft_consensus.cc:348] T 00000000000000000000000000000000 P cba31bc073e647d7b48fe47ea5d71c41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cba31bc073e647d7b48fe47ea5d71c41" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42407 } }
I20240503 02:01:25.337750 23580 raft_consensus.cc:374] T 00000000000000000000000000000000 P cba31bc073e647d7b48fe47ea5d71c41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:01:25.337822 23580 raft_consensus.cc:729] T 00000000000000000000000000000000 P cba31bc073e647d7b48fe47ea5d71c41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cba31bc073e647d7b48fe47ea5d71c41, State: Initialized, Role: FOLLOWER
I20240503 02:01:25.338243 23580 consensus_queue.cc:260] T 00000000000000000000000000000000 P cba31bc073e647d7b48fe47ea5d71c41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cba31bc073e647d7b48fe47ea5d71c41" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42407 } }
I20240503 02:01:25.338347 23580 raft_consensus.cc:388] T 00000000000000000000000000000000 P cba31bc073e647d7b48fe47ea5d71c41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:01:25.338383 23580 raft_consensus.cc:482] T 00000000000000000000000000000000 P cba31bc073e647d7b48fe47ea5d71c41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:01:25.338467 23580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cba31bc073e647d7b48fe47ea5d71c41 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:01:25.339300 23580 raft_consensus.cc:504] T 00000000000000000000000000000000 P cba31bc073e647d7b48fe47ea5d71c41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cba31bc073e647d7b48fe47ea5d71c41" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42407 } }
I20240503 02:01:25.339646 23580 leader_election.cc:302] T 00000000000000000000000000000000 P cba31bc073e647d7b48fe47ea5d71c41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cba31bc073e647d7b48fe47ea5d71c41; no voters:
I20240503 02:01:25.339936 23580 leader_election.cc:288] T 00000000000000000000000000000000 P cba31bc073e647d7b48fe47ea5d71c41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:01:25.340071 23585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cba31bc073e647d7b48fe47ea5d71c41 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:01:25.340665 23585 raft_consensus.cc:686] T 00000000000000000000000000000000 P cba31bc073e647d7b48fe47ea5d71c41 [term 1 LEADER]: Becoming Leader. State: Replica: cba31bc073e647d7b48fe47ea5d71c41, State: Running, Role: LEADER
I20240503 02:01:25.340926 23585 consensus_queue.cc:237] T 00000000000000000000000000000000 P cba31bc073e647d7b48fe47ea5d71c41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cba31bc073e647d7b48fe47ea5d71c41" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42407 } }
I20240503 02:01:25.340967 23580 sys_catalog.cc:564] T 00000000000000000000000000000000 P cba31bc073e647d7b48fe47ea5d71c41 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:01:25.342499 23586 sys_catalog.cc:455] T 00000000000000000000000000000000 P cba31bc073e647d7b48fe47ea5d71c41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cba31bc073e647d7b48fe47ea5d71c41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cba31bc073e647d7b48fe47ea5d71c41" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42407 } } }
I20240503 02:01:25.342700 23586 sys_catalog.cc:458] T 00000000000000000000000000000000 P cba31bc073e647d7b48fe47ea5d71c41 [sys.catalog]: This master's current role is: LEADER
I20240503 02:01:25.342497 23587 sys_catalog.cc:455] T 00000000000000000000000000000000 P cba31bc073e647d7b48fe47ea5d71c41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cba31bc073e647d7b48fe47ea5d71c41. Latest consensus state: current_term: 1 leader_uuid: "cba31bc073e647d7b48fe47ea5d71c41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cba31bc073e647d7b48fe47ea5d71c41" member_type: VOTER last_known_addr { host: "127.0.99.190" port: 42407 } } }
I20240503 02:01:25.342767 23587 sys_catalog.cc:458] T 00000000000000000000000000000000 P cba31bc073e647d7b48fe47ea5d71c41 [sys.catalog]: This master's current role is: LEADER
I20240503 02:01:25.343062 23599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:01:25.346431 23599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:01:25.351917 23599 catalog_manager.cc:1347] Generated new cluster ID: 5507055acba44aaa8b3f96ba2b3d53a0
I20240503 02:01:25.352041 23599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:01:25.360430 23599 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:01:25.361392 23599 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:01:25.379251 23599 catalog_manager.cc:5857] T 00000000000000000000000000000000 P cba31bc073e647d7b48fe47ea5d71c41: Generated new TSK 0
I20240503 02:01:25.380175 23599 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:01:25.395361 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.129:0
--local_ip_for_outbound_sockets=127.0.99.129
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42407
--builtin_ntp_servers=127.0.99.148:39877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 02:01:25.518206 23604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:01:25.518488 23604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:01:25.518600 23604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:01:25.522933 23604 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:01:25.523069 23604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:01:25.523200 23604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.129
I20240503 02:01:25.528540 23604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.99.129
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42407
--never_fsync=true
--heap_profile_path=/tmp/kudu.23604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.129
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:01:25.529991 23604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:01:25.531647 23604 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:01:25.544013 23609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:01:25.544143 23612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:01:25.544584 23610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:01:25.545886 23604 server_base.cc:1034] running on GCE node
I20240503 02:01:25.546404 23604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:01:25.547140 23604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:01:25.548350 23604 hybrid_clock.cc:648] HybridClock initialized: now 1714701685548329 us; error 37 us; skew 500 ppm
I20240503 02:01:25.550689 23604 webserver.cc:460] Webserver started at http://127.0.99.129:40521/ using document root <none> and password file <none>
I20240503 02:01:25.551314 23604 fs_manager.cc:359] Metadata directory not provided
I20240503 02:01:25.551384 23604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:01:25.551589 23604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:01:25.553416 23604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "883db30ce7584d67bcb83a892909924e"
format_stamp: "Formatted at 2024-05-03 02:01:25 on dist-test-slave-w9tw"
I20240503 02:01:25.554070 23604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "883db30ce7584d67bcb83a892909924e"
format_stamp: "Formatted at 2024-05-03 02:01:25 on dist-test-slave-w9tw"
I20240503 02:01:25.557922 23604 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20240503 02:01:25.560619 23618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:01:25.562168 23604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 02:01:25.562284 23604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "883db30ce7584d67bcb83a892909924e"
format_stamp: "Formatted at 2024-05-03 02:01:25 on dist-test-slave-w9tw"
I20240503 02:01:25.562381 23604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:01:25.579779 23604 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:01:25.580471 23604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:01:25.580643 23604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:01:25.581164 23604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:01:25.582105 23604 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:01:25.582176 23604 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:01:25.582221 23604 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:01:25.582237 23604 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:01:25.593719 23604 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.129:35539
I20240503 02:01:25.593771 23731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.129:35539 every 8 connection(s)
I20240503 02:01:25.594791 23604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:01:25.602710 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 23604
I20240503 02:01:25.602943 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:01:25.606235 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.130:0
--local_ip_for_outbound_sockets=127.0.99.130
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42407
--builtin_ntp_servers=127.0.99.148:39877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:01:25.606844 23732 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42407
I20240503 02:01:25.607197 23732 heartbeater.cc:464] Registering TS with master...
I20240503 02:01:25.608188 23732 heartbeater.cc:511] Master 127.0.99.190:42407 requested a full tablet report, sending...
I20240503 02:01:25.610643 23544 ts_manager.cc:194] Registered new tserver with Master: 883db30ce7584d67bcb83a892909924e (127.0.99.129:35539)
I20240503 02:01:25.612504 23544 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.129:40317
W20240503 02:01:25.738173 23735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:01:25.738451 23735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:01:25.738554 23735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:01:25.741948 23735 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:01:25.742039 23735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:01:25.742137 23735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.130
I20240503 02:01:25.749183 23735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.99.130
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42407
--never_fsync=true
--heap_profile_path=/tmp/kudu.23735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.130
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:01:25.750370 23735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:01:25.752172 23735 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:01:25.761463 23740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:01:25.762493 23741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:01:25.763236 23743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:01:25.764372 23735 server_base.cc:1034] running on GCE node
I20240503 02:01:25.764966 23735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:01:25.765607 23735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:01:25.766769 23735 hybrid_clock.cc:648] HybridClock initialized: now 1714701685766749 us; error 44 us; skew 500 ppm
I20240503 02:01:25.769342 23735 webserver.cc:460] Webserver started at http://127.0.99.130:37121/ using document root <none> and password file <none>
I20240503 02:01:25.769948 23735 fs_manager.cc:359] Metadata directory not provided
I20240503 02:01:25.770018 23735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:01:25.770215 23735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:01:25.772090 23735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "2f81f2f8f0ef43d79855645893371024"
format_stamp: "Formatted at 2024-05-03 02:01:25 on dist-test-slave-w9tw"
I20240503 02:01:25.772600 23735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "2f81f2f8f0ef43d79855645893371024"
format_stamp: "Formatted at 2024-05-03 02:01:25 on dist-test-slave-w9tw"
I20240503 02:01:25.776826 23735 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 02:01:25.779587 23749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:01:25.780862 23735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:01:25.781021 23735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "2f81f2f8f0ef43d79855645893371024"
format_stamp: "Formatted at 2024-05-03 02:01:25 on dist-test-slave-w9tw"
I20240503 02:01:25.781136 23735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:01:25.808379 23735 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:01:25.809361 23735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:01:25.809563 23735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:01:25.810168 23735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:01:25.811200 23735 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:01:25.811257 23735 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:01:25.811322 23735 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:01:25.811353 23735 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:01:25.821604 23735 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.130:33665
I20240503 02:01:25.821794 23862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.130:33665 every 8 connection(s)
I20240503 02:01:25.824051 23735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:01:25.826473 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 23735
I20240503 02:01:25.826730 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:01:25.832444 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.131:0
--local_ip_for_outbound_sockets=127.0.99.131
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42407
--builtin_ntp_servers=127.0.99.148:39877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:01:25.835410 23863 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42407
I20240503 02:01:25.835695 23863 heartbeater.cc:464] Registering TS with master...
I20240503 02:01:25.836210 23863 heartbeater.cc:511] Master 127.0.99.190:42407 requested a full tablet report, sending...
I20240503 02:01:25.837368 23544 ts_manager.cc:194] Registered new tserver with Master: 2f81f2f8f0ef43d79855645893371024 (127.0.99.130:33665)
I20240503 02:01:25.837980 23544 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.130:38105
W20240503 02:01:25.971470 23866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:01:25.971752 23866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:01:25.971799 23866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:01:25.975111 23866 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:01:25.975208 23866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:01:25.975349 23866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.131
I20240503 02:01:25.979825 23866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.99.131
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42407
--never_fsync=true
--heap_profile_path=/tmp/kudu.23866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.131
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:01:25.980865 23866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:01:25.982293 23866 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:01:25.995922 23871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:01:25.995947 23872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:01:25.997145 23874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:01:25.998323 23866 server_base.cc:1034] running on GCE node
I20240503 02:01:25.998878 23866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:01:25.999523 23866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:01:26.000754 23866 hybrid_clock.cc:648] HybridClock initialized: now 1714701686000719 us; error 52 us; skew 500 ppm
I20240503 02:01:26.003202 23866 webserver.cc:460] Webserver started at http://127.0.99.131:38889/ using document root <none> and password file <none>
I20240503 02:01:26.003830 23866 fs_manager.cc:359] Metadata directory not provided
I20240503 02:01:26.003916 23866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:01:26.004128 23866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:01:26.005982 23866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a6d9dbf064624b49805fb9b1b9bedcb1"
format_stamp: "Formatted at 2024-05-03 02:01:26 on dist-test-slave-w9tw"
I20240503 02:01:26.006489 23866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a6d9dbf064624b49805fb9b1b9bedcb1"
format_stamp: "Formatted at 2024-05-03 02:01:26 on dist-test-slave-w9tw"
I20240503 02:01:26.010233 23866 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240503 02:01:26.013280 23880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:01:26.014578 23866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240503 02:01:26.014722 23866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a6d9dbf064624b49805fb9b1b9bedcb1"
format_stamp: "Formatted at 2024-05-03 02:01:26 on dist-test-slave-w9tw"
I20240503 02:01:26.014829 23866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:01:26.028892 23866 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:01:26.029573 23866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:01:26.029750 23866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:01:26.030306 23866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:01:26.031304 23866 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:01:26.031363 23866 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:01:26.031425 23866 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:01:26.031453 23866 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:01:26.040174 23866 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.131:43757
I20240503 02:01:26.040217 23993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.131:43757 every 8 connection(s)
I20240503 02:01:26.041297 23866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:01:26.050427 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 23866
I20240503 02:01:26.050848 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:01:26.052860 23994 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42407
I20240503 02:01:26.053112 23994 heartbeater.cc:464] Registering TS with master...
I20240503 02:01:26.053237 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.132:0
--local_ip_for_outbound_sockets=127.0.99.132
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42407
--builtin_ntp_servers=127.0.99.148:39877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:01:26.053654 23994 heartbeater.cc:511] Master 127.0.99.190:42407 requested a full tablet report, sending...
I20240503 02:01:26.054733 23544 ts_manager.cc:194] Registered new tserver with Master: a6d9dbf064624b49805fb9b1b9bedcb1 (127.0.99.131:43757)
I20240503 02:01:26.055318 23544 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.131:43115
W20240503 02:01:26.172215 23997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:01:26.172498 23997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:01:26.172583 23997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:01:26.175824 23997 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:01:26.175928 23997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:01:26.176038 23997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.132
I20240503 02:01:26.180073 23997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.99.132
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42407
--never_fsync=true
--heap_profile_path=/tmp/kudu.23997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.132
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:01:26.181126 23997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:01:26.182283 23997 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:01:26.192071 24003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:01:26.193358 24005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:01:26.193879 24002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:01:26.195003 23997 server_base.cc:1034] running on GCE node
I20240503 02:01:26.195503 23997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:01:26.196128 23997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:01:26.197306 23997 hybrid_clock.cc:648] HybridClock initialized: now 1714701686197259 us; error 65 us; skew 500 ppm
I20240503 02:01:26.199508 23997 webserver.cc:460] Webserver started at http://127.0.99.132:35405/ using document root <none> and password file <none>
I20240503 02:01:26.200098 23997 fs_manager.cc:359] Metadata directory not provided
I20240503 02:01:26.200165 23997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:01:26.200352 23997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:01:26.202064 23997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "9dfff02fbacf44299911c36eaa2e1c2b"
format_stamp: "Formatted at 2024-05-03 02:01:26 on dist-test-slave-w9tw"
I20240503 02:01:26.202603 23997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "9dfff02fbacf44299911c36eaa2e1c2b"
format_stamp: "Formatted at 2024-05-03 02:01:26 on dist-test-slave-w9tw"
I20240503 02:01:26.206451 23997 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240503 02:01:26.209478 24011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:01:26.211319 23997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240503 02:01:26.211503 23997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "9dfff02fbacf44299911c36eaa2e1c2b"
format_stamp: "Formatted at 2024-05-03 02:01:26 on dist-test-slave-w9tw"
I20240503 02:01:26.211622 23997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:01:26.243350 23997 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:01:26.244045 23997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:01:26.244230 23997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:01:26.244802 23997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:01:26.245743 23997 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:01:26.245796 23997 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:01:26.245854 23997 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:01:26.245889 23997 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:01:26.255299 23997 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.132:36845
I20240503 02:01:26.255317 24124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.132:36845 every 8 connection(s)
I20240503 02:01:26.256306 23997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:01:26.260991 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 23997
I20240503 02:01:26.261107 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:01:26.263495 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.133:0
--local_ip_for_outbound_sockets=127.0.99.133
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42407
--builtin_ntp_servers=127.0.99.148:39877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:01:26.266976 24125 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42407
I20240503 02:01:26.267221 24125 heartbeater.cc:464] Registering TS with master...
I20240503 02:01:26.267761 24125 heartbeater.cc:511] Master 127.0.99.190:42407 requested a full tablet report, sending...
I20240503 02:01:26.269104 23544 ts_manager.cc:194] Registered new tserver with Master: 9dfff02fbacf44299911c36eaa2e1c2b (127.0.99.132:36845)
I20240503 02:01:26.269675 23544 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.132:39261
W20240503 02:01:26.388929 24128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:01:26.389263 24128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:01:26.389333 24128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:01:26.392657 24128 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:01:26.392736 24128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:01:26.392812 24128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.133
I20240503 02:01:26.397061 24128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.99.133
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42407
--never_fsync=true
--heap_profile_path=/tmp/kudu.24128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.133
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:01:26.398155 24128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:01:26.399401 24128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:01:26.412070 24133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:01:26.412328 24136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:01:26.412829 24134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:01:26.414259 24128 server_base.cc:1034] running on GCE node
I20240503 02:01:26.414830 24128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:01:26.415522 24128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:01:26.416756 24128 hybrid_clock.cc:648] HybridClock initialized: now 1714701686416697 us; error 78 us; skew 500 ppm
I20240503 02:01:26.419126 24128 webserver.cc:460] Webserver started at http://127.0.99.133:33279/ using document root <none> and password file <none>
I20240503 02:01:26.419760 24128 fs_manager.cc:359] Metadata directory not provided
I20240503 02:01:26.419823 24128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:01:26.419992 24128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:01:26.421871 24128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "b62bdd85e5da40c39cb0b849c9ea7d5c"
format_stamp: "Formatted at 2024-05-03 02:01:26 on dist-test-slave-w9tw"
I20240503 02:01:26.422367 24128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "b62bdd85e5da40c39cb0b849c9ea7d5c"
format_stamp: "Formatted at 2024-05-03 02:01:26 on dist-test-slave-w9tw"
I20240503 02:01:26.426609 24128 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240503 02:01:26.429994 24142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:01:26.431576 24128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:01:26.431716 24128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b62bdd85e5da40c39cb0b849c9ea7d5c"
format_stamp: "Formatted at 2024-05-03 02:01:26 on dist-test-slave-w9tw"
I20240503 02:01:26.431841 24128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:01:26.451272 24128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:01:26.452030 24128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:01:26.452204 24128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:01:26.452800 24128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:01:26.453754 24128 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:01:26.453804 24128 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:01:26.453848 24128 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:01:26.453883 24128 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:01:26.463567 24128 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.133:38847
I20240503 02:01:26.463717 24255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.133:38847 every 8 connection(s)
I20240503 02:01:26.464578 24128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:01:26.471766 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 24128
I20240503 02:01:26.471916 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 02:01:26.474427 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.134:0
--local_ip_for_outbound_sockets=127.0.99.134
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42407
--builtin_ntp_servers=127.0.99.148:39877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:01:26.474598 24256 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42407
I20240503 02:01:26.474790 24256 heartbeater.cc:464] Registering TS with master...
I20240503 02:01:26.475276 24256 heartbeater.cc:511] Master 127.0.99.190:42407 requested a full tablet report, sending...
I20240503 02:01:26.476320 23544 ts_manager.cc:194] Registered new tserver with Master: b62bdd85e5da40c39cb0b849c9ea7d5c (127.0.99.133:38847)
I20240503 02:01:26.477119 23544 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.133:44497
W20240503 02:01:26.606045 24259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:01:26.606359 24259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:01:26.606408 24259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:01:26.610440 24259 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:01:26.610642 24259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:01:26.610766 24259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.134
I20240503 02:01:26.615370 23732 heartbeater.cc:503] Master 127.0.99.190:42407 was elected leader, sending a full tablet report...
I20240503 02:01:26.615684 24259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.99.134
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42407
--never_fsync=true
--heap_profile_path=/tmp/kudu.24259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.134
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:01:26.617518 24259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:01:26.619021 24259 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:01:26.630561 24264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:01:26.631569 24265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:01:26.631726 24267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:01:26.632879 24259 server_base.cc:1034] running on GCE node
I20240503 02:01:26.633374 24259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:01:26.634050 24259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:01:26.635232 24259 hybrid_clock.cc:648] HybridClock initialized: now 1714701686635211 us; error 36 us; skew 500 ppm
I20240503 02:01:26.638387 24259 webserver.cc:460] Webserver started at http://127.0.99.134:42971/ using document root <none> and password file <none>
I20240503 02:01:26.639243 24259 fs_manager.cc:359] Metadata directory not provided
I20240503 02:01:26.639335 24259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:01:26.639551 24259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:01:26.641872 24259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "c944b0a61617419386ec653fa47e7505"
format_stamp: "Formatted at 2024-05-03 02:01:26 on dist-test-slave-w9tw"
I20240503 02:01:26.642894 24259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "c944b0a61617419386ec653fa47e7505"
format_stamp: "Formatted at 2024-05-03 02:01:26 on dist-test-slave-w9tw"
I20240503 02:01:26.647526 24259 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240503 02:01:26.651222 24273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:01:26.653193 24259 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240503 02:01:26.653390 24259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "c944b0a61617419386ec653fa47e7505"
format_stamp: "Formatted at 2024-05-03 02:01:26 on dist-test-slave-w9tw"
I20240503 02:01:26.653504 24259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:01:26.670569 24259 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:01:26.671277 24259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:01:26.671439 24259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:01:26.672035 24259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:01:26.673084 24259 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:01:26.673144 24259 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:01:26.673213 24259 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:01:26.673249 24259 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:01:26.683100 24259 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.134:34397
I20240503 02:01:26.683149 24386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.134:34397 every 8 connection(s)
I20240503 02:01:26.684423 24259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:01:26.692523 398 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu as pid 24259
I20240503 02:01:26.692665 398 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 02:01:26.695478 398 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
/tmp/dist-test-taskSkLu3v/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.99.135:0
--local_ip_for_outbound_sockets=127.0.99.135
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42407
--builtin_ntp_servers=127.0.99.148:39877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 02:01:26.696731 24387 heartbeater.cc:347] Connected to a master server at 127.0.99.190:42407
I20240503 02:01:26.697003 24387 heartbeater.cc:464] Registering TS with master...
I20240503 02:01:26.697664 24387 heartbeater.cc:511] Master 127.0.99.190:42407 requested a full tablet report, sending...
I20240503 02:01:26.699098 23544 ts_manager.cc:194] Registered new tserver with Master: c944b0a61617419386ec653fa47e7505 (127.0.99.134:34397)
I20240503 02:01:26.699981 23544 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.99.134:52607
W20240503 02:01:26.831679 24390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:01:26.831952 24390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:01:26.832000 24390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:01:26.835688 24390 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 02:01:26.835783 24390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:01:26.835861 24390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.99.135
I20240503 02:01:26.840503 23863 heartbeater.cc:503] Master 127.0.99.190:42407 was elected leader, sending a full tablet report...
I20240503 02:01:26.841441 24390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.99.148:39877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.99.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.99.135
--webserver_port=0
--tserver_master_addrs=127.0.99.190:42407
--never_fsync=true
--heap_profile_path=/tmp/kudu.24390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.99.135
--log_dir=/tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type DEBUG
built by None at 03 May 2024 01:43:14 UTC on 1a15035a6830
build id 97
I20240503 02:01:26.843628 24390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:01:26.845137 24390 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:01:26.855808 24395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:01:26.856907 24396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:01:26.857932 24398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:01:26.857898 24390 server_base.cc:1034] running on GCE node
I20240503 02:01:26.858409 24390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:01:26.859156 24390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:01:26.860358 24390 hybrid_clock.cc:648] HybridClock initialized: now 1714701686860316 us; error 59 us; skew 500 ppm
I20240503 02:01:26.863270 24390 webserver.cc:460] Webserver started at http://127.0.99.135:36379/ using document root <none> and password file <none>
I20240503 02:01:26.864243 24390 fs_manager.cc:359] Metadata directory not provided
I20240503 02:01:26.864404 24390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:01:26.864814 24390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:01:26.867425 24390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "2c2eb83e5ef94b77bce0621f455bcd22"
format_stamp: "Formatted at 2024-05-03 02:01:26 on dist-test-slave-w9tw"
I20240503 02:01:26.868034 24390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSkLu3v/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714701525263522-398-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "2c2eb83e5ef94b77bce0621f455bcd22"
format_stamp: "Formatted at 2024-05-03 02:01:26 on dist-test-slave-w9tw"